oversight

Audit of the USAJOBS System Development Lifecycle FY 2012

Published by the Office of Personnel Management, Office of Inspector General on 2012-09-28.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                                     U.S. OFFICE OF PERSONNEL MANAGEMENT
                                                           OFFICE OF THE INSPECTOR GENERAL
                                                                            OFFICE OF AUDITS




                                   Final Audit Report

Subject:




                    AUDIT OF THE USAJOBS
               SYSTEM DEVELOPMENT LIFECYCLE
                           FY 2012
                                         Report No. 4A-HR-00-12-044


                                         Date:                 ____________    ___
                                                               September 28, 2012




                                                          --CAUTION--
This audit report has been distributed to Federal officials who are responsible for the administration of the audited program. This audit
report may contain proprietary data which is protected by Federal law (18 U.S.C. 1905). Therefore, while this audit report is available
under the Freedom of Information Act and made available to the public on the OIG webpage, caution needs to be exercised before
releasing the report to the general public as it may contain proprietary information that was redacted from the publicly distributed copy.
                                                       Audit Report

                              U.S. OFFICE OF PERSONNEL MANAGEMENT
                               -------------------------------------------------------------

                                         AUDIT OF THE USAJOBS
                                    SYSTEM DEVELOPMENT LIFECYCLE
                                                FY 2012

                                                  --------------------------------
                                                    WASHINGTON, D.C.




                                         Report No. 4A-HR-00-12-044


                                                                9/28/12




                                                                                                      Date:   ____________ ___




                                                                                     ______________________
                                                                                     Michael R. Esser
                                                                                     Assistant Inspector General
                                                                                        for Audits
                                                          --CAUTION--
This audit report has been distributed to Federal officials who are responsible for the administration of the audited program. This audit
report may contain proprietary data which is protected by Federal law (18 U.S.C. 1905). Therefore, while this audit report is available
under the Freedom of Information Act and made available to the public on the OIG webpage, caution needs to be exercised before
releasing the report to the general public as it may contain proprietary information that was redacted from the publicly distributed copy.
                                   Executive Summary


                     U.S. OFFICE OF PERSONNEL MANAGEMENT
                      -------------------------------------------------------------

                              AUDIT OF THE USAJOBS
                         SYSTEM DEVELOPMENT LIFECYCLE
                                     FY 2012

                                    --------------------------------
                                      WASHINGTON, D.C.




                              Report No. 4A-HR-00-12-044


                               Date:            ____________
                                                9/28/12      __


The objectives of this audit were to assess the system development lifecycle (SDLC)
methodology of USAJOBS and to determine if any lessons learned from the USAJOBS 3.0
deployment could be applied to future system implementation projects at the U.S. Office of
Personnel Management (OPM). OPM has been historically plagued with failed and troubled
system implementation projects, and we believe that weak SDLC practices have played a major
role in this.

Our audit evaluated SDLC elements such as requirements gathering, infrastructure change
management, application change management, and testing. We looked at both the controls that
were in place at the time of system deployment in October 2011, and also the controls that have
been implemented and improved in the nine months since deployment.

Although our audit revealed some specific weaknesses in the original USAJOBS SDLC and
some recommendations to improve current procedures, we believe that the overall methodology
has improved significantly and that the system is operating with a stable change management
process.


                                                   i
Our primary concern relates to the fact that the entire USAJOBS SDLC methodology was
developed independent of any agency-wide requirements or guidance – because no current
guidance exists at OPM. Although OPM’s internal website contains policies and procedures
related to SDLC, many of these documents have not been updated in over 10 years, and they are
not routinely used to manage current development projects.

After reviewing our draft audit report, the Office of the Chief Information Officer (OCIO)
notified us of recent and ongoing efforts to create a current SDLC policy. While we
acknowledge that creating a policy is a significant first step in implementing a centralized SDLC
methodology at OPM, the policy will need additional updating in order to address the specific
deficiencies identified in this report. In addition, policy alone will not improve the historically
weak SDLC management capabilities of OPM.

We recommend that the OCIO establish an SDLC review process in which the OCIO must
review and formally approve SDLC work at various milestones for all OPM system
implementation projects. All of our audit recommendations related to a centralized SDLC
program at OPM should remain open until this process has been fully implemented and evidence
can be produced to indicate that the new policies are actively enforced.

In addition to our concerns about OPM's overall SDLC management, this audit discovered the
following controls in place and opportunities for improvement specific to the USAJOBS system:

•   We reviewed system requirements of USAJOBS and determined that they were well
    documented and organized. Nothing came to our attention to indicate that there were any
    deficiencies in the OCIO’s requirements gathering methodology for USAJOBS.

•   The OCIO generally has good controls related to infrastructure change management.
    However, we were unable to independently verify that all infrastructure changes were
    formally approved. We also determined that the OCIO has not yet implemented a process to
    routinely audit the actual configuration of its servers to ensure that they are compliant with
    the approved baseline.

•   The OCIO has implemented a thorough change management process to facilitate changes to
    the USAJOBS application. However, we noticed an inconsistency in the way change
    requests were approved and recommend that the OCIO develop a policy that outlines what
    individuals can make formal approvals at various stages in the USAJOBS application change
    management process.

•   Prior to its deployment, USAJOBS 3.0 was subject to rigorous testing from a variety of
    sources. However, the test environment available in the weeks prior to deployment did not
    have the full set of data that would be loaded to the production environment. OPM
    experienced great difficulty in cleanly transferring the data from the old Monster
    Government Solutions (MGS) system to the new USAJOBS 3.0. These difficulties were
    driven by the weak contract language that did not require MGS to provide OPM with the
    system details that would facilitate a more graceful transition of data.


                                                 ii
•   Most of the issues experienced in the first week after the deployment of USAJOBS 3.0 were
    related to an unprecedented number of users stressing the system’s resources. The OCIO
    provided us with evidence indicating that they did perform a variety of stress tests on
    USAJOBS prior to launch. However, the system was unable to handle the unprecedented
    number of users that attempted to access the system once it went live. We believe that the
    OCIO should analyze and document the lessons learned from this experience and apply them
    toward future system development projects at OPM.

•   The testing process for USAJOBS has consistently improved since the system’s deployment
    and is now functioning adequately.




                                              iii
                                                                 Contents
                                                                                                                                               Page

Executive Summary ......................................................................................................................... i
Introduction and Background ..........................................................................................................1
Objectives ........................................................................................................................................1
Scope and Methodology ..................................................................................................................1
Compliance with Laws and Regulations..........................................................................................2
Results ..............................................................................................................................................3
   A.         SDLC Overview ............................................................................................................... 3
   B.         Requirements Gathering................................................................................................... 4
   C.         Infrastructure Configuration and Change Management ................................................... 5
   D.         Application Change Management .................................................................................... 6
   E.         Testing .............................................................................................................................. 7
Major Contributors to this Report ..................................................................................................12
Appendix: The Office of the Chief Information Officer's August 7, 2012 response to the draft
          audit report, issued July 18, 2012.
                          Introduction and Background

USAJOBS is the federal government’s official one-stop source for Federal jobs and employment
information. The USAJOBS website provides public notice of Federal employment
opportunities to Federal employees and United States citizens. USAJOBS is cooperatively
owned by the federal Chief Human Capital Officer (CHCO) council.

In 2003, OPM contracted with Monster Government Services (MGS) to host and maintain the
USAJOBS system. In 2010, the Office of Personnel Management (OPM) and the CHCO
Council made the decision to not renew its contract with MGS and to bring USAJOBS in-house
at OPM. One element of this decision was based on the fact that two separate security breaches
at MGS led to the disclosure of sensitive USAJOBS data.

In October 2011, OPM launched USAJOBS 3.0. This new version of USAJOBS was developed
by various members of the CHCO council with primary contributions from OPM, the
Department of Homeland Security, and the Department of Defense. USAJOBS 3.0 is hosted at
OPM’s data center in Macon, Georgia and is maintained by two divisions of OPM’s Office of
the Chief Information Officer (OCIO): the application business owners – USAJOBS Program
Office, and the development and technical infrastructure support team – Human Resources Tools
and Technology (HRTT.)

When USAJOBS 3.0 was deployed, the system became flooded with an unprecedented number
of users trying to access the public website. The system’s communications lines did not have the
bandwidth to manage the traffic and many users experienced a variety of errors that resulted
from dropped network communications, or were unable to access the system altogether. These
issues led to a public outcry from the media and by the general population via the USAJOBS
social networking websites. Furthermore, the House of Representatives Committee on Oversight
and Government Reform questioned the OPM Director about the agency’s ability to manage
large information system development projects.

                                        Objectives
The objectives of this audit were to assess the SDLC methodology of USAJOBS and to
determine if any lessons learned from the USAJOBS 3.0 deployment could be applied to future
OPM system implementation projects. These objectives were met by reviewing the following
elements of the USAJOBS project:

•   Requirements Gathering;
•   Infrastructure Change Management;
•   Application Change Management; and,
•   Testing.

                               Scope and Methodology
This performance audit was conducted by the Office of the Inspector General (OIG) in
accordance with Government Auditing Standards, issued by the Comptroller General of the


                                               1
United States. Accordingly, the audit included an evaluation of related policies and procedures,
compliance tests, and other auditing procedures that we considered necessary. The audit
documented the controls in place for USAJOBS as of July 2012.

We considered the USAJOBS internal control structure in planning our audit procedures. These
procedures were mainly substantive in nature, although we did gain an understanding of
management procedures and controls to the extent necessary to achieve our audit objectives.

Our audit evaluated SDLC elements such as requirements gathering, infrastructure change
management, application change management, and testing. We looked at both the controls that
were in place at the time of system deployment in October 2011, and also the controls that have
been implemented and improved in the nine months since deployment.

In conducting the audit, we relied to varying degrees on computer-generated data. Due to time
constraints, we did not verify the reliability of the data generated by the various information
systems involved. However, nothing came to our attention during our audit testing utilizing the
computer-generated data to cause us to doubt its reliability. We believe that the data was
sufficient to achieve the audit objectives. Except as noted above, the audit was conducted in
accordance with generally accepted government auditing standards issued by the Comptroller
General of the United States.

Details of our audit findings and recommendations are located in the “Results” section of this
report. Since our audit would not necessarily disclose all significant matters in the internal
control structure, we do not express an opinion on the USAJOBS system of internal controls
taken as a whole.

The audit was conducted from February through July 2012 in OPM’s Washington, D.C.
headquarters building.

                        Compliance with Laws and Regulations
In conducting the audit, we performed tests to determine whether OPM’s management of
USAJOBS is consistent with applicable standards. Nothing came to our attention during this
review to indicate that OPM is in violation of relevant laws and regulations.




                                                2
                                           Results
The sections below provide a summary of our audit findings and recommendations related to the
SDLC of USAJOBS and OPM’s overall SDLC methodology.

A. SDLC Overview

   We reviewed the USAJOBS SDLC to verify that the OCIO has implemented adequate
   controls to ensure that the system continues to operate smoothly and to prevent reoccurrences
   of the problems that occurred in the first few days after the system was deployed. OPM has
   been historically plagued with failed and troubled system implementation projects, and we
   believe that weak SDLC practices have played a major role in this.

   Our audit evaluated SDLC elements such as requirements gathering, infrastructure change
   management, application change management, and testing. We looked at both the controls
   that were in place at the time of system deployment in October 2011, and also the controls
   that have been implemented and improved in the nine months since deployment.

   Although the sections below detail some specific weaknesses in the original USAJOBS
   SDLC and some recommendations to improve current procedures, we believe that the overall
   methodology has improved significantly and that the system is operating with a stable change
   management process.

   Our primary concern relates to the fact that the entire USAJOBS SDLC methodology was
   developed independent of any agency-wide requirements or guidance – because no current
   guidance exists at OPM. Although OPM’s internal website contains policies and procedures
   related to SDLC, many of these documents have not been updated in over 10 years, and they
   are not routinely used to manage current development projects. System development at OPM
   has become a decentralized process managed by the individual program offices that own and
   operate information systems. Our audits of these various projects have revealed significant
   inconsistencies in the methodology and quality of SDLC management.

   We believe that the OCIO needs to develop current policies and procedures that outline the
   minimum requirements of critical SDLC components. The OCIO should also take an active
   oversight role in all systems development projects in the agency, and establish a formal
   SDLC review team that must review SDLC work at various milestones or checkpoints and
   formally approve the project to move forward.

   Recommendation 1
   We recommend that the OCIO develop an agency-wide SDLC methodology with specific
   policies and procedures that must be followed for all system development projects at OPM.
   The policies and requirements should consider the various approaches to system
   implementation (build-from-scratch, commercial software, etc.) routinely used by OPM.




                                               3
   OCIO Response:
   “The Office of CIO has updated the Information Technology Systems Manager (ITSM)
   standards to reflect an agency-wide system development life cycle (SDLC) methodology.
   The update is called ‘OPM System Development Life Cycle Policy and Standards.’ The
   policy document applies to all OPM programs with an IT component, regardless of
   funding type and amount. It is to be used in conjunction with ITSM templates and will
   replace other ITSM documentation and addresses various approaches to system
   implementation routinely used by OPM . . . . This document has completed final reviews
   and is undergoing Web Team preparation to be published on the agency public website
   (www.opm.gov) in the near future. The templates, which are to be used with it, are
   currently located on THEO at http://theo.opm.gov/itsm/Templ.asp.”

   OIG Reply:
   We agree that the new OPM System Development Life Cycle Policy and Standards
   document is a significant first step in implementing a centralized SDLC methodology at
   OPM. However, the policy will need additional updating in order to address the specific
   deficiencies identified in this report. Additionally, policy alone will not improve the
   historically weak SDLC management capabilities of OPM. This recommendation should
   remain open until the SDLC review process (see Recommendation 2) has been fully
   implemented and evidence can be produced to indicate that the new policies are actively
   enforced.

   Recommendation 2
   We recommend that the OCIO establish an SDLC review process in which the OCIO must
   review and formally approve SDLC work at various milestones for all OPM system
   implementation projects. The minimum elements that the OIG believes should be
   incorporated into this review process are detailed in Recommendations 3, 7, and 8, below.

   OCIO Response:
   “We will review the new SDLC Policy and Standards document and the templates
   described above to identify appropriate responsibility for approval of SDLC work at
   various milestones.”

   OIG Reply:
   In addition to identifying appropriate personnel to approve SDLC work at various
   milestones, the OCIO should update the SDLC policy to provide details of these milestones
   and the requirements and deliverables for each.

B. Requirements Gathering

   After the decision was made to in-source USAJOBS, OPM faced the task of documenting the
   functional requirements of the system. Due to weak language in the original contract with
   MGS, OPM did not have access to the source code, database schemas, data values, tables,
   etc., of the existing USAJOBS system operated by MGS. Therefore, engineers in the OCIO
   had to reverse-engineer the functional elements of the system to document its requirements.




                                              4
   Using the Agile system development lifecycle approach, the developers and the business
   owners worked together to develop specific functional requirements in the form of “user
   stories.”

   We reviewed the original set of user stories and determined that the original requirements of
   USAJOBS appeared to be well documented. Nothing came to our attention to indicate that
   there were any deficiencies in the OCIO’s requirements gathering methodology for
   USAJOBS.

   However, the methodology successfully used by the USAJOBS program office was
   implemented by the system’s developers and business owners and was independent of any
   agency-wide policy, procedures, or guidance. We have reviewed a variety of failed and
   troubled systems implementation projects at OPM and have often found that poor
   requirements gathering and documentation contributed to the failure.

   Recommendation 3
   As part of the recommended SDLC review process, we recommend that the OCIO develop a
   policy that provides guidance on requirements gathering for new information systems and
   outlines minimum documentation requirements.

   OCIO Response:
   “Please see the new SDLC Policy and Standards document, attached.”

   OIG Reply:
   We acknowledge the fact that the new SDLC Policy addresses requirements gathering.
   However, the policy should be updated to outline the requirements and deliverables related to
   this milestone in the SDLC process. This recommendation should remain open until the
   SDLC review process (see Recommendation 2) has been fully implemented and evidence can
   be produced to indicate that the new requirements gathering policies are actively enforced.

C. Infrastructure Configuration and Change Management

   The OCIO generally has good controls related to infrastructure change management.
   However, we did note two opportunities for improvement in this area.

   The OCIO maintains a detailed inventory of the computer hardware that supports the
   USAJOBS system infrastructure, and has developed a detailed baseline configuration that
   outlines a standard secure configuration for both application and web servers.

   All changes to the approved configuration have been documented for all USAJOBS servers.
   However, we were unable to independently verify that all changes were formally approved.
   We selected a sample of USAJOBS infrastructure changes and asked the OCIO for evidence
   that these changes were approved. The OCIO’s response indicated that many of the changes
   were approved verbally or via informal e-mail. Although we have no reason to believe that
   these changes were not verbally approved, the OCIO should begin to formally document this
   communication so that there is an auditable trail of approval activity.



                                               5
   In addition, the OCIO has not yet implemented a process to routinely audit the actual
   configuration of its servers to ensure that they are compliant with the approved baseline.
   Routine configuration audits would alert the OCIO of any changes that were made outside of
   the standard change management process.

   Recommendation 4
   We recommend that the OCIO develop and implement a procedure to formally document
   approvals for USAJOBS infrastructure changes (changes made to server configurations).

   OCIO Response:
   “On July 30, the USAJOBS Configuration Management Plan was updated to outline
   formal approvals for USAJOBS infrastructure changes. Specifically, future changes to
   server configurations will be approved in writing by the Chief, Systems Capacity Branch
   (SCB), HRTT. The records of these approvals will be stored with the HRTT SCB.”

   OIG Reply:
   As part of the audit resolution process, we recommend that the OCIO provide OPM’s
   Internal Oversight and Compliance Office (IOC) with evidence that the Configuration
   Management Plan was updated and that the subsequent infrastructure changes were approved
   in writing.

   Recommendation 5
   We recommend that the OCIO develop and implement a procedure to routinely audit the
   actual configuration of the USAJOBS servers and compare the settings to the approved
   baseline configuration.

   OCIO Response:
   “A thorough annual review of the USAJOBS configuration is conducted by the USAJOBS
   Designated Security Officer (DSO) and the HRTT SCB as required by the HRTT
   Information Technology (IT) Security Standard Operating Procedure (SOP). The DSO
   also receives and reviews a monthly report of the servers, software versions, and
   configurations. The USAJOBS Configuration Management Plan has been updated to
   include this review activity for USAJOBS configuration changes and comparison with
   approved baseline configurations.”

   OIG Reply:
   As part of the audit resolution process, we recommend that the OCIO provide IOC with
   evidence that the Configuration Management Plan was updated and that a configuration audit
   has been conducted.

D. Application Change Management

   The OCIO has implemented a thorough change management process to facilitate changes to
   the USAJOBS application. A software product,                                 is used to
   manage system requirements and the status of all changes to the application. contains




                                             6
   the details of all existing features of the system and also a “backlog” of fixes and
   enhancements that are being developed for future releases.

   Both the developers (HRTT) and the business owners (USAJOBS Program Office) have
   access to      and both use this product to facilitate real-time communication on the status
   of individual work items.      is also used to track the various approvals that are required
   throughout the application change process.

   We selected a sample of application changes and viewed the history of these items within
         All changes in the sample were subject to formal approvals within        However, we
   did notice an inconsistency in the way these items were approved. Some work items were
   approved by the business owners and others were approved by individuals that worked on the
   development staff. The OCIO explained that none of the developers that actually worked on
   coding a work item were involved in approving that change (which would be a conflict of
   interest). Although the OCIO’s explanations of these anomalies seems reasonable, there is
   no formal policy describing who can approve various types of application changes, and we
   were therefore unable to independently verify that these approvals were appropriate.

   Recommendation 6
   We recommend that the OCIO develop a policy that outlines which individuals can make
   formal approvals at various stages in the USAJOBS application change management process.

   OCIO Response:
   “While there was a standard operating procedure in place, it was not formally
   documented. On July 27, the USAJOBS Release Management SOP was updated to address
   the steps performed in      to track development work as it moves from one stage of the
   process through the next. It outlines which approvals are represented and who is required
   to perform the action.”

   OIG Reply:
   As part of the audit resolution process, we recommend that the OCIO provide IOC with
   evidence that the Release Management SOP was updated to address this recommendation.

E. Testing

   We evaluated the OCIO’s methodology for testing USAJOBS prior to its deployment and
   also the testing process currently in place today.

   Pre-deployment functionality testing

   Prior to its deployment, USAJOBS 3.0 was subject to rigorous testing from a variety of
   sources. The OCIO maintains evidence that the system was tested by developers, business
   owners, users, and also by external vendors whose systems interface with USAJOBS.

   All pre-deployment test plans had passed before the system went live. However, the test
   environment available in the weeks prior to deployment did not have the full set of data that



                                                 7
would be loaded to the production environment. OPM experienced great difficulty in cleanly
transferring the data from the old MGS system to the new USAJOBS 3.0. These difficulties
were driven by the weak contract language that did not require MGS to provide OPM with
the system details that would facilitate a more graceful transition of data. Therefore, most of
the pre-deployment testing occurred in a test environment that, while fully functional, did not
have all of the data that would be present in production.

As a result, pre-deployment tests could not reveal all anomalies in the system. This was a
particular problem for the testing of location codes (i.e., the search engine’s ability to
recognize abbreviations and alternate spellings of locations and provide accurate results).
For example, test searches for Ft. Meade, MD and Fort Meade, Maryland may not produce
consistent results because the limited test environment data didn’t include any job postings
from that area. The full set of clean data was not loaded to the system until just before the
deployment date, and the OCIO did not have time to start the testing process over. Delaying
the release of the system to conduct further testing would have cost OPM $500,000 per
month in contract extension fees with MGS.

Although no current audit recommendations can address the problems that occurred with
USAJOBS, we believe that the OCIO should take steps to prevent testing related issues from
occurring in future system development projects.

Recommendation 7
As part of the recommended SDLC checkpoint process, we recommend that the OCIO
implement a policy that provides general guidance and minimum requirements for pre-
deployment testing. The policy should also require all new systems to undergo testing in a
fully functional test environment with a full set of data prior to system launch.

OCIO Response:
“See the new SDLC Policy and Standards document, attached. It addresses testing
requirements. (See, for example, section 4.2.5 ‘Build System Components Phase’, p. 23,
and Appendix D.4, p. 86 – 97. See also Appendix D.2, ‘Define System Requirements Phase
Activities’, p. 66 – 78.) Such sections provide general guidance, including checklists of
activities for testing. We will evaluate the new SDLC Policy and Standards document, and
will consider other options as well, to determine the best approach for establishing
minimum requirements for pre-deployment testing.”

OIG Reply:
We acknowledge the fact that the new SDLC Policy addresses system testing at a high level.
However, the policy should be updated to outline the requirements and deliverables for
testing-related milestones in the SDLC process. This recommendation should remain open
until the SDLC review process (see Recommendation 2) has been fully implemented and
evidence can be produced to indicate that the testing requirements are being actively
enforced.




                                             8
Pre-deployment stress testing

Most of the issues experienced in the first week after the deployment of USAJOBS 3.0 were
related to an unprecedented number of users stressing the system’s resources. The OCIO
provided us with evidence indicating that they did perform a variety of stress tests on
USAJOBS prior to launch. The system was able to successfully process a traffic load that
simulated the busiest day on USAJOBS under the prior operator.

However, the system was unable to handle the unprecedented number of users that attempted
to access the system once it went live. Although the servers and databases were not
operating at capacity, the communications lines did not have the bandwidth necessary to
manage the traffic. As a result, users experienced a variety of errors that resulted from
dropped packets or were unable to access the system altogether.

Another issue that added stress to the system was the fact that every USAJOBS user was
required to change their password upon first login to the new USAJOBS 3.0 system. This
was a result of MGS not having to transfer existing password data to OPM (see reference to
weak contract language in section A, above).

Within a week of the system’s deployment, OPM contracted with a content delivery network
solution provider whose services drastically reduced the stress on OPM’s communication
lines. USAJOBS is now operating at about 10-12% capacity on the communications lines.

The system is now stable and no current audit recommendation would be relevant to
USAJOBS stress testing. However, in hindsight it is easy to recognize the variables that led
to the unprecedented traffic that USAJOBS experienced (for example: the advertisement of a
“new jobs site” in a weak economy, the fact that users were unable to access the system for
almost a week prior to launch, and search engine spiders exploring and archiving the new
website.) We believe that the OCIO should analyze and document the lessons learned from
this experience and apply them toward future system development projects at OPM.

Recommendation 8
As part of the recommended SDLC checkpoint process, we recommend that the OCIO
develop a policy that outlines the minimum requirements for stress testing of a new
information system.

OCIO Response
“Please see the new SDLC Policy and Standards document, attached. As noted in response
to Recommendation 7, above, it addresses testing requirements, and provides general
guidance and checklists of activities for testing. We will evaluate the new SDLC Policy
and Standards document, and will consider other options as well, to determine the best
approach for establishing minimum requirements for stress testing of new information
systems.”




                                           9
OIG Reply:
We acknowledge the fact that the new SDLC Policy addresses system testing at a high level.
However, the policy should be updated to outline the requirements and deliverables for
testing-related milestones in the SDLC process. This recommendation should remain open
until the SDLC review process (see Recommendation 2) has been fully implemented and
evidence can be produced to indicate that the testing requirements are being actively
enforced.

Current testing process

We evaluated the OCIO’s procedures for testing post-deployment changes to USAJOBS by
reviewing testing documentation for all modifications made to USAJOBS since its initial
release. Although portions of the testing process were inconsistent and not well documented
in the first months after the system’s deployment, we believe that the testing methodology
has consistently improved and is now functioning adequately.

All changes to the USAJOBS application are subject to testing from both the development
(HRTT) and the business owner (USAJOBS program office) sides. Each side has its own
unique testing methodology. The program office testing methodology has been consistent
and well documented since the beginning of the USAJOBS 3.0 project, and we were able to
review detailed test scripts and results for every change. However, the testing process for the
HRTT developers has evolved since the initial release of USAJOBS 3.0.

While we have no reason to doubt that HRTT has tested all post-deployment changes to
USAJOBS, the testing activity was poorly documented for early changes to the system.
There are several changes where no testing-related documentation exists (testing activity was
communicated verbally), and others where testing was documented via informal e-mails
simply stating “the test passed.” In addition, these early changes were not tested with
formally documented test scripts.

HRTT has recently implemented a software package that helps it manage change testing
activity. This software allows the developers to document a detailed test script complete
with expected results. The system also allows the developers to mark items as “passed” once
they have been successfully tested, thereby creating an auditable record of testing activity.

We reviewed the completed test plan for the latest release of updates to USAJOBS.
Although we did not detect any anomalies in the recent testing documentation we believe
that, since this process is relatively new, it should be subject to further monitoring to ensure
that it is functioning appropriately. We also believe that the OCIO should formalize and
document its now-stable testing methodology to ensure that all future changes are tested and
documented consistently.

Recommendation 9
We recommend that the OCIO provide IOC with the developer test plans and documented
results for the next two releases/updates of USAJOBS.




                                             10
OCIO Response
“We will provide test documentation for Release 3.3 and 3.4 upon completion of 3.4 and
deployment by August 31, 2012.”

Recommendation 10
We recommend that the OCIO develop a testing policy for USAJOBS that outlines all of the
elements that need to be documented for all testing activity (test plans, test scripts, results,
etc.)

OCIO Response
“The USAJOBS Program Office drafted this policy for the program in February 2012,
however, it was never completed. The USAJOBS Program Office and HRTT will jointly
work together to update our Testing Plan to specifically outline testing artifacts, activities,
and the location of these records for audit purposes. We estimate that we can complete this
activity by December 31, 2012.”




                                             11
                          Major Contributors to this Report
This audit report was prepared by the U.S. Office of Personnel Management, Office of Inspector
General, Information Systems Audits Group. The following individuals participated in the audit
and the preparation of this report:

•                  , Group Chief
•                    , Senior Team Leader




                                              12
CIO Response: The new SDLC Policy and Standards document, mentioned above, describes
phases and methods and identifies responsibility for approval of many SDLC products. (Unlike
the ITSM, which did not address Agile process, the new SDLC Policy and Standards document
requires that for Agile (Scrum) methodology there be Stage Gate Reviews of monthly milestones
so that performance can be determined. See Appendix F.6 of the new SDLC document.)

We will review the new SDLC Policy and Standards document and the templates described
above to identify appropriate responsibility for approval of SDLC work at various milestones.

Recommendation 3 states, “As part of the recommended SDLC review process, we recommend
that the OCIO develop a policy that provides guidance on requirements gathering for new
information systems and outlines minimum documentation requirements.”

CIO Response: Please see the new SDLC Policy and Standards document, attached.

Infrastructure Configuration and Change Management
Recommendation 4 asked that the Office of the Chief Information Officer (OCIO) develop and
implement a procedure to formally document approvals for USAJOBS infrastructure changes
(changes made to server configurations).

CIO Response: On July 30, the USAJOBS Configuration Management Plan was updated to
outline formal approvals for USAJOBS infrastructure changes. Specifically, future changes to
server configurations will be approved in writing by the Chief, Systems Capacity Branch (SCB),
HRTT. The records of these approvals will be stored with the HRTT SCB.

Recommendation 5 recommended that the OCIO develop and implement a procedure to
routinely audit the actual configuration of the USAJOBS servers and compare the settings to the
approved baseline configuration.

CIO Response: A thorough annual review of the USAJOBS configuration is conducted by the
USAJOBS Designated Security Officer (DSO) and the HRTT SCB as required by the HRTT
Information Technology (IT) Security Standard Operating Procedure (SOP). The DSO also
receives and reviews a monthly report of the servers, software versions, and configurations. The
USAJOBS Configuration Management Plan has been updated to include this review activity for
USAJOBS configuration changes and comparison with approved baseline configurations.

Application Change Management
Recommendation 6 recommended that the OCIO develop a policy that outlines what
individuals can make formal approvals at various stages in the USAJOBS application change
management process.

CIO Response: While there was a standard operating procedure in place, it was not formally
documented. On July 27, the USAJOBS Release Management SOP was updated to address the
steps performed in        to track development work as it moves from one stage of the process
through the next. It outlines which approvals are represented and who is required to perform the
action.

                                                2
We appreciate continued support of the USAJOBS program and the CIO SDLC initiatives.

Attachment

      -    OPM System Development Life Cycle Policy and Standards, v. 1.0, June 2012




cc:
          Director, Integrated Hiring Systems
          Office of the Chief Information Officer


          Chief, IT Investment Management
          Office of the Chief Information Officer


          Chief, Information Security and Privacy
          Office of the Chief Information Officer


          Director
          Internal Oversight and Compliance




                                                    4