oversight

DOJ Workforce Planning: Grant-Making Components Should Enhance the Utility of Their Staffing Models

Published by the Government Accountability Office on 2012-12-14.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                United States Government Accountability Office

GAO             Report to Congressional Committees




                DOJ WORKFORCE
December 2012



                PLANNING

                Grant-Making
                Components Should
                Enhance the Utility of
                Their Staffing Models




GAO-13-92
                                               December 2012

                                               DOJ WORKFORCE PLANNING
                                               Grant-Making Components Should Enhance the
                                               Utility of Their Staffing Models
Highlights of GAO-13-92, a report to
congressional committees




Why GAO Did This Study                         What GAO Found
DOJ annually awards billions of dollars        The Department of Justice’s (DOJ) three grant-making components—the Office
to thousands of grant recipients, which        of Justice Programs (OJP), the Office of Community Oriented Policing Services
presents administrative and workload           (COPS), and the Office on Violence Against Women (OVW)—have partially met
challenges for DOJ’s workforce. In May         five of six leading practices, and fully met another, to ensure that the staffing
2011, DOJ’s grant-making components            models the contractor prepared for each of them to assess their workloads and
began deploying new staffing models            workforce capacities are sound and reliable. These six leading practices are
to analyze their workloads and better          broad areas that represent a summary of the activities that should take place
ensure that staffing levels and                during the three stages of model design, development, and deployment. For
personnel assignments reflect their
                                               example, the components partially met the leading practice for ensuring the
responsibilities. In November 2011, the
                                               credibility of data used in the models. This practice includes both verifying with
conference report accompanying
DOJ’s FY 2012 appropriation directed
                                               staff that the models accurately depict the component’s operations—an activity in
GAO to evaluate each component’s               which they fully engaged—and testing the models for logic and accuracy—an
staffing model. In response, this report       activity in which they partially engaged. Specifically, the components did not
addresses (1) the extent to which the          comprehensively check all of the models’ formulas, and GAO found errors with
grant-making components incorporated           all three components’ models’ retirement calculations. When left uncorrected,
leading practices to ensure that their         such errors have implications for workforce-related decisions and, ultimately, can
staffing models are sound and reliable         affect the efficiency and effectiveness of component operations. In contrast, the
and (2) the components’ use of the             components fully met one leading practice and its associated activities for
models to inform their workforce               leveraging stakeholders. For example, all three components engaged grant
planning efforts, and the extent to            managers and senior program officials when working with the contractor to apply
which they used the models for budget          their knowledge and experience to design and develop the models. Moving
development. GAO reviewed the                  forward, fully incorporating leading practices when updating the models for future
staffing models the components                 use will better position the grant-making components to capitalize on the models’
procured under contract in 2011, and           capacity and facilitate effective workforce planning and budgeting.
the updated, 2012 versions; identified
six leading practices based on our prior       Due to the deficiencies GAO identified in the design, development, and
work for ensuring the models’                  deployment of the staffing models, grant-making components have not used the
soundness and reliability; and                 models to identify workforce gaps or to inform the preparation of annual budget
assessed the components’ actions for           requests. In particular, the three grant-making components have not used the
meeting each practice. GAO also                models to determine critical skills and competencies of staff and identify related
interviewed component officials about          gaps. Further, officials from all three components stated that they would like to,
their use of the models.                       or plan to, use the models to inform the upcoming fiscal year 2015 budget
                                               requests, but they do not yet have a strategy for doing so. For example, officials
What GAO Recommends                            from all three grant-making components recognize that their human capital staff
                                               will need to share and interpret the data in the models with their budget staff, but
GAO recommends that DOJ
                                               as of late September 2012, no time frames, coordination principles, or guidance
components fully incorporate leading
                                               were in place to ensure this exchange of information. Standard practices for
practices to help ensure their staffing
models’ soundness and reliability, and         project management, including budget development, involve the establishment of
develop and implement a strategy for           timelines, coordination principles, and guidance. Developing, documenting, and
using the models to inform workforce           implementing a strategy that includes these elements could help the grant-
planning and budget development.               making components better use the models to inform budget development. In
DOJ agreed with these                          particular, such a strategy should account for the full incorporation of leading
recommendations and reported it                practices to ensure the models are sound and reliable and that the quantifiable
would take actions to address them.            data from the models provide the proper context and justification to the Congress
                                               for the funding levels the components request.
View GAO-13-92. For more information,
contact David C. Maurer at (202) 512-9627 or
maurerd@gao.gov.

                                                                                        United States Government Accountability Office
Contents


Letter                                                                                     1
               Background                                                                  6
               Grant-Making Components Have At Least Partially Met Leading
                 Practices to Ensure that the Staffing Models Are Sound and
                 Reliable, But Can Strengthen Their Efforts                              15
               Grant-Making Components Have Not Used the Staffing Models to
                 Determine Workforce Gaps and Do Not Have a Strategy for
                 Using the Models to Inform Budget Requests                              29
               Conclusions                                                               35
               Recommendations for Executive Action                                      36
               Agency Comments                                                           36

Appendix I     Objectives, Scope, and Methodology                                        38



Appendix II    Comments from the Department of Justice                                   41



Appendix III   GAO Contact and Staff Acknowledgments                                     45



Tables
               Table 1: Overview of Active Grants, Authorized Positions, and
                        Onboard Positions, by Grant-Making Component from
                        Fiscal Years 2008 through 2011                                    9
               Table 2: Four Phases of the Federal Grant Life Cycle                      10
               Table 3: Key Contractor Deliverables                                      14
               Table 4: Leading Practices for Ensuring Model Soundness and
                        Reliability                                                      16
               Table 5: Extent to Which Grant-Making Components Took Actions
                        to Account for Risk Associated with Contracting Out
                        Model Development                                                16
               Table 6: Extent to Which Grant-Making Components Took Actions
                        to Ensure the Credibility of Data Used in the Models             18
               Table 7: Extent to Which Grant-Making Components Took Actions
                        to Preserve the Integrity of Data Maintained in the Models       21
               Table 8: Extent to Which Grant-Making Components Took Actions
                        to Establish Roles and Responsibilities for Staff Assigned
                        to Update and Operate the Models                                 23



               Page i                                        GAO-13-92 DOJ Workforce Planning
          Table 9: Timeliness of Components’ Designation of Responsible
                   Staff to Initiate Updates to the Models                                          24
          Table 10: Extent to Which Grant-Making Components Took
                   Actions to Ensure Adequate Training for Staff Assigned to
                   Update and Operate the Models                                                    26
          Table 11: Extent to Which Grant-Making Components Took
                   Actions to Leverage Stakeholders’ Knowledge and
                   Experience in Designing and Developing the Models                                27


Figures
          Figure 1: Department of Justice Grant Funding Appropriated from
                   Fiscal Years 2008 through 2011                                                   8
          Figure 2: Timeline of Events That Affected the Grant-Making
                   Components’ Plans to Expand the Size of their
                   Workforces                                                                       13




          Abbreviations
          ACOR        Assistant Contracting Officer Representative
          COPS        Community Oriented Policing Services
          COR         Contracting Officer Representative
          DOJ         Department of Justice
          OJP         Office of Justice Programs
          OVW         Office on Violence Against Women


          This is a work of the U.S. government and is not subject to copyright protection in the
          United States. The published product may be reproduced and distributed in its entirety
          without further permission from GAO. However, because this work may contain
          copyrighted images or other material, permission from the copyright holder may be
          necessary if you wish to reproduce this material separately.




          Page ii                                                GAO-13-92 DOJ Workforce Planning
United States Government Accountability Office
Washington, DC 20548




                                   December 14, 2012

                                   The Honorable Barbara Mikulski
                                   Chairwoman
                                   The Honorable Kay Bailey Hutchison
                                   Ranking Member
                                   Subcommittee on Commerce, Justice,
                                     Science, and Related Agencies
                                   Committee on Appropriations
                                   United States Senate

                                   The Honorable Frank Wolf
                                   Chairman
                                   The Honorable Chaka Fattah
                                   Ranking Member
                                   Subcommittee on Commerce, Justice,
                                     Science, and Related Agencies
                                   Committee on Appropriations
                                   House of Representatives

                                   At the close of fiscal year 2012, the Department of Justice’s (DOJ) three
                                   grant-making components—the Office of Justice Programs (OJP), the
                                   Office of Community Oriented Policing Services (COPS Office), and the
                                   Office on Violence Against Women (OVW)—were responsible for
                                   overseeing just over $13 billion in federal financial assistance to support
                                   crime prevention, law enforcement, and crime victim services to about
                                   31,000 grantees operating at the state and local levels. 1 The fiscal year
                                   2012 enacted budgets of these three components comprise about 9
                                   percent of DOJ’s total budget authority, and to manage their workload,
                                   the three grant-making components are authorized for about 1 percent of



                                   1
                                    DOJ provides federal financial assistance pursuant to statutory authorization. The
                                   assistance typically takes the form of formula grants; discretionary grants; and cooperative
                                   agreements and all are generally referred to as grants. Formula grant programs are
                                   noncompetitive awards based on a predetermined formula, typically established in statute.
                                   Discretionary grants are usually awarded on the basis of a competitive selection process.
                                   A cooperative agreement is a type of grant wherein the federal government is more
                                   substantially involved than usual in carrying out associated activities. For more
                                   information, see GAO, Justice Grant Programs: DOJ Should Do More to Reduce the Risk
                                   of Unnecessary Duplication and Enhance Program Assessment, GAO-12-517
                                   (Washington, D.C., Jul. 12, 2012).




                                   Page 1                                                  GAO-13-92 DOJ Workforce Planning
DOJ’s total staffing level—a total of 960 positions. Staff members within
these offices are involved in all aspects of grant awarding, including
preparing funding announcements to describe the allowable uses of new
grant funds, disbursing the awards, and routinely monitoring grantees’
compliance with financial, administrative, and programmatic
requirements. 2 In addition to this workload, the grant-making component
workforce has other responsibilities as well, including but not limited to
providing technical assistance to grantees, conducting research or
preparing publications, and responding to external audits. To ensure that
the grant-making components’ respective staffing levels and personnel
assignments accurately reflect the responsibilities and requirements of
each office, DOJ’s Justice Management Division (JMD) awarded a $2
million contract to a private sector consulting company to develop a
workload and workforce planning model (i.e., “a staffing model”) for each
of the three grant-making components. 3 Performance on the contract
began in December 2010, and in May 2011, the contractor delivered a
staffing model to each granting-making component based on common
principles and assumptions, yet unique to their respective business
processes. In addition, the contractor delivered a number of related
planning documents, such as suggested strategies for developing
workforce skills.

We have previously reported on the need for strategic workforce planning
to enhance performance and ensure accountability in federally funded
programs. 4 Further, we have designated strategic human capital
management as a high-risk area and reported that the government’s
management of its people—its human capital—was the critical missing
link in reforming and modernizing the federal government’s management




2
GAO-12-517.
3
 Among other responsibilities, JMD helps ensure strategic and human capital planning
and oversees budget development for all of the agencies within DOJ.
4
 GAO, Human Capital: Key Principles for Effective Strategic Workforce Planning,
GAO-04-39 (Washington, D.C.: Dec. 11, 2003).




Page 2                                                GAO-13-92 DOJ Workforce Planning
practices. 5 As we have also previously reported, the number of grant
programs and recipients, and the billions of dollars in funds DOJ awards
annually, present administrative and workload challenges for DOJ’s
workforce. 6

In November 2011, the conference report accompanying DOJ’s
appropriations act for fiscal year 2012 directed DOJ to detail actual and
estimated costs of grant-making components’ management and
administrative expenses, by grant program, in future budget
submissions. 7 The costs are to include categories of expenses other than
grants, such as providing technical assistance and conducting research
and statistics, among others. The conferees specifically encouraged the
grant-making components to minimize administrative spending in order to
maximize the amount of funding that can be used for grants. In addition,
the conference report acknowledged the grant-making components’
efforts to analyze their workloads, and directed us to evaluate each grant-
making component’s staffing model and make recommendations, as
warranted, on how each model could be further improved. This report
addresses the following questions: (1) to what extent have the grant-
making components incorporated leading practices to ensure that their
staffing models are sound and reliable?, and (2) how have the
components used the models to inform their workforce planning efforts
and to what extent have the components begun to use the models for
budget development?

To address the first question, we assessed how the staffing models are
structured and the steps the grant-making components have taken to
ensure that the models are sound and reliable. We primarily reviewed our




5
 GAO, High Risk Series: An Update, GAO-01-263 (Washington, D.C.: Jan.1, 2001). Since
that time, the federal government has made progress in addressing its human capital
challenges and we have therefore narrowed the scope of this high-risk area to focus on
the most significant challenges that remain to close current and emerging critical skills
gaps in vital areas such as acquisition, foreign language capabilities, and oil and gas
management. See GAO, High Risk Series: An Update, GAO-11-278 (Washington, D.C.:
Feb. 16, 2011).
6
GAO-12-517.
7
H.R. Rep. No. 112-284, at 242-43 (2011) (Conf. Rep.)




Page 3                                                 GAO-13-92 DOJ Workforce Planning
prior work 8 to identify six leading practices for ensuring the staffing
models’ soundness (i.e., their validity and reliability): (1) accounting for
risk associated with contracting out model development, (2) ensuring the
credibility of data used in the models, (3) preserving the integrity of data
maintained in the models, (4) establishing roles and responsibilities for
staff assigned to update and operate the models, (5) ensuring adequate
training for staff assigned to update and operate the models, and (6)
leveraging stakeholders’ knowledge and experience in designing and
developing the models. We then assessed each individual grant-making
component’s reported actions against the actions associated with each
leading practice and determined the extent to which the practices were
met. To do this, we used a scale of “fully met,” “partially met,” and “not
met.” A determination of “fully met” means that the grant-making
component provided evidence that it had completed all actions associated
with the leading practice. A determination of “partially met” means that the
grant-making component provided evidence that it had completed some
actions associated with the leading practice. A determination of “not met”
means that the grant-making component provided no evidence that it had
completed any actions associated with the leading practice.

We also interviewed grant-making components’ senior officials with
responsibility for grant administration as well as for human capital
management and budget planning to understand how they worked with
the contractor to design and develop the staffing models. We also
interviewed those in each grant-making component responsible for
updating and executing their respective models to get perspectives on the
training they received, their prior workforce planning backgrounds, and


8
 We primarily used GAO, Internal Control Management and Evaluation Tool,
GAO-01-1008G (Washington, D.C.: Aug. 1, 2001) to develop the six leading practices.
These leading practices are also informed and supported by GAO, Designing Evaluations:
2012 Revision, GAO-12-208G (Washington, D.C.: Jan. 31, 2012); Federal Information
System Controls Audit Manual, GAO-09-232G (Washington, D.C.: Feb. 2009); and
Assessing the Reliability of Computer-Processed Data, GAO-09-680G (Washington, D.C.:
Jul. 2009). GAO has also examined the concepts of soundness and reliability in the
following reports: See GAO, Tax Administration: IRS Needs to Strengthen Its Approach for
Evaluating the SRFMI Data-Sharing Pilot Program, GAO-09-45 (Washington, D.C.: Nov.
7, 2008); Catastrophic Planning: States Participating in FEMA’s Pilot Program Made
Progress, but Better Guidance Could Enhance Future Pilot Programs, GAO-11-383
(Washington, D.C.: Apr. 8, 2011); Telecommunications: FCC’s Performance Management
Weaknesses Could Jeopardize Proposed Reforms of the Rural Health Care
Program,GAO-11-27 (Washington, D.C.: Nov. 17, 2010); and Equal Employment
Opportunity: Pilot Projects Could Help Test Solutions to Long-standing Concerns with the
EEO Complaint Process, GAO-09-712 (Washington, D.C.: Aug. 12, 2009).




Page 4                                               GAO-13-92 DOJ Workforce Planning
the time they spent working with the models. In addition, we analyzed
documentation about the grant-making components’ practices related to
updating the models, as well as the contractor’s guidelines for doing so.
We then compared the grant-making components’ practices for
designing, developing, and deploying the models for use to the leading
practices we identified and made judgments about the extent to which
they aligned. We also assessed each grant-making component’s model
independently, reviewing all of the contractor’s deliverables and
understanding the formulas and inputs constituting the models’
framework. To evaluate each model, we used a discovery method
approach, wherein we examined a small number of strategically drawn
formulas and calculations. While we were not attempting to develop a
specific error rate per se, this discovery method approach permitted
conclusions about the risk for (a) additional errors in other parts of the
model we did not examine, and (b) potential errors in the model’s outputs.

To address the second question, we interviewed grant-making
component and JMD officials on their workforce planning efforts prior to
having the staffing models and discussed how the components are
conducting workforce planning since obtaining the models and the ways
in which they are using the models to inform their activities. We also
discussed the grant-making components’ efforts and future plans to
integrate the models into their budget submissions and the role JMD
plays in facilitating these efforts. In addition, we reviewed related
documents to support the officials’ testimonial evidence and identified
relevant practices from our prior work for effectively linking budgets and
workforce plans and compared DOJ’s activities and plans with these
practices. More details on our scope and methods appear in Appendix I.

We conducted this performance audit from May 2012 to December 2012
in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain
sufficient, appropriate evidence to provide a reasonable basis for our
findings and conclusions based on our audit objectives. We believe that
the evidence obtained provides a reasonable basis for our findings and
conclusions based on our audit objectives.




Page 5                                        GAO-13-92 DOJ Workforce Planning
Background
Workforce Planning   Workforce planning is a process for getting the right number of people,
Concepts             with the right skills, in the right job, at the right time to accomplish an
                     organization’s mission and goals. As we have reported previously,
                     strategic workforce planning addresses two critical needs: (1) aligning an
                     organization’s human capital management efforts with its current and
                     emerging mission and programmatic goals, and (2) developing long-term
                     strategies for acquiring, developing, and retaining staff to achieve
                     programmatic goals. 9

                     At its heart, workforce planning involves assessing workload;
                     understanding the skills and limitations (i.e., capacity) of the existing
                     workforce; and addressing the gaps needed to optimize performance.
                     While organizations’ approaches to workforce planning will vary, our
                     earlier work established five key principles that strategic workforce
                     planning should address, irrespective of the context in which the planning
                     is done. These include the following:

                     •   involving top management, employees, and other stakeholders in
                         developing, communicating, and implementing the strategic workforce
                         plan;

                     •   setting a strategic vision to determine the critical skills and
                         competencies that will be needed to achieve current and future
                         programmatic results;

                     •   identifying and developing strategies that are tailored to address gaps
                         in number, deployment, and alignment of human capital approaches
                         for enabling and sustaining the contributions of all critical skills and
                         competencies;

                     •   building the capability needed to address administrative, educational,
                         and other requirements important to support workforce planning
                         strategies; and




                     9
                     GAO-04-39.




                     Page 6                                        GAO-13-92 DOJ Workforce Planning
                        •     monitoring, evaluating, and revising the organization’s progress
                              toward its human capital goals and the contribution that human capital
                              results have made toward achieving programmatic results.

                        In addition, we have previously reported on the importance of ensuring
                        that data used to make key decisions, such as for workforce planning, are
                        both sound and reliable—meaning that the data are accurate, credible,
                        and appropriate for analysis. 10


Grant-Making            DOJ administers its grant programs through three grant-making
Components’ Roles,      components—OJP, the COPS Office, and OVW.
Responsibilities, and
                        •     OJP is comprised of a number of bureaus and offices and its mission
Funding                       is to develop the nation’s capacity to prevent and control crime,
                              administer justice, and assist crime victims. As such, it administers
                              federal financial assistance to state and local governments, for-profit
                              and nonprofit organizations, tribal jurisdictions, and educational
                              institutions to help address victim assistance, technology and
                              forensics, and juvenile justice, among other things.

                        •     The COPS Office advances the practice of community policing, which
                              supports the systematic use of partnerships and problem-solving
                              techniques to address public safety issues such as crime, social
                              disorder, and fear of crime. The COPS Office makes its grants
                              available to police departments, including tribal law enforcement
                              components, as well as researchers. Across its programs, the COPS
                              Office has also funded schools and other organizations promoting
                              public safety.

                        •     OVW seeks to provide federal leadership in developing the nation’s
                              capacity to reduce violence against women and strengthen services to
                              victims of domestic violence, dating violence, sexual assault, and
                              stalking. Like OJP and the COPS Office, its grant funding is available
                              to an array of recipients, including those providing legal assistance,
                              tribal and local law enforcement components, schools and campuses,
                              and other advocacy organizations.




                        10
                            GAO-09-45 and GAO-11-383.




                        Page 7                                          GAO-13-92 DOJ Workforce Planning
The levels of appropriations across all three grant-making components
have varied over time, as figure 1 illustrates, but OJP has consistently
received the greatest amount. In fiscal year 2011, OJP had approximately
$2.5 billion in funding for grants; the COPS Office had approximately
$528 million; and OVW had approximately $431 million.

Figure 1: Department of Justice Grant Funding Appropriated from Fiscal Years 2008
through 2011




Notes: (1) Appropriations figures include American Recovery and Reinvestment Act (ARRA), Pub. L.
No. 111-5, 123 Stat. 115, funds but do not include carryover from prior years or payment programs,
which are managed differently from typical grants in that they generally fund grantees on a
reimbursable rather than direct basis. OJP is the only grant-making component to operate payment
programs. (2) COPS Office appropriations reflect budget authority that has been provided to the
COPS Office by DOJ’s annual appropriation acts, such as the DNA Backlog Reduction Program,
even though these appropriations are shifted to OJP through an internal transfer that gives OJP
program administration responsibilities. The amounts were $243,084,000 in 2008; $272,000,000 in
2009; $203,000,000 in 2010; and, $168,153,020 in 2011.


Similarly, the number of grants in active status, number of authorized
positions, and the number of onboard positions across the grant-making




Page 8                                                      GAO-13-92 DOJ Workforce Planning
components have varied over time, as shown in table 1, but OJP has
consistently been the largest component. 11 Specifically, OJP managed
more than 19,000 grants, and had 702 authorized positions and 668
onboard positions in fiscal year 2011. 12 In contrast, OVW, the smallest of
the three, managed more than 2,800 grants and had 70 authorized
positions with 61 onboard positions.

Table 1: Overview of Active Grants, Authorized Positions, and Onboard Positions,
by Grant-Making Component from Fiscal Years 2008 through 2011

 OJP                                            2008            2009            2010             2011
 Active grants                                14,912          18,899          19,438          19,404
 Authorized positions                            697              697             702             702
 Onboard positions                               637              690             696             668


 COPS Office                                    2008            2009            2010             2011
 Active grants                                 4,614           4,505            4,844           4,401
 Authorized positions                            166              166             188             188
 Onboard positions                               108              116             134             136


 OVW                                            2008            2009            2010             2011
 Active grants                                 2,533           2,578            2,685           2,871
 Authorized positions                              65              65              65              70
 Onboard positions                                 48              58              66              61
Source: OJP, COPS Office, and OVW.

Notes: (1) Active grants are those that were active for more than 1 day during the fiscal year, and do
not reflect the agency’s entire workload. (2) Authorized positions include full- and part-time federal
employees but do not include additional contractor staff. There are some authorized positions whose
responsibilities are not directly related to grants management yet whose duties are critical to the
operation of the component. (3) The level of actual, onboard personnel may be lower at any given
time than the total positions authorized, and may also be slightly higher so long as DOJ is within the
allowable, authorized position cap departmentwide.


The federal financial assistance the grant-making components administer
typically takes the form of discretionary grants, formula grants, and


11
  Authorized positions include full- and part-time federal employees but do not include
additional contractor staff. There are some authorized positions whose responsibilities are
not directly related to grants management yet whose duties are critical to the operation of
the component.
12
     Onboard positions refer to personnel that are actually in place at any given time.




Page 9                                                         GAO-13-92 DOJ Workforce Planning
cooperative agreements, and all are generally referred to as grants. 13 The
grants that OJP, the COPS Office, and OVW award generally follow a
similar life cycle, including pre-award, award, implementation, and closeout
stages, as table 2 illustrates. OJP and OVW use a web-based system—the
Grants Management System (GMS)—to manage this process, while the
COPS Office uses a separate COPS Management System (CMS), which is
a desktop, integrated set of programmatic, financial, and related
applications that communicates with a centralized database.

Table 2: Four Phases of the Federal Grant Life Cycle

    Grant phase      Common activities
    Pre-award        • Announce the grant opportunity with a grant solicitation, which
                       notifies potential applicants of the grant’s purpose, the terms of the
                       award, the amount available, and the eligibility criteria, among
                       other information
                     • Receive and review applications
                     • Make award decisions based on requirements set by statute and
                       the grant-making component, as appropriate
    Award            • Notify the grantee
                     • Publicly announce the awarding of funds

    Implementation   • Disburse payment
                     • Collect and review grantee progress reports to check that grants
                       are being implemented as intended and that grantees are
                       compliant with statutory or regulatory requirements as well as any
                       applicable policy guidelines
                     • Conduct site visits to monitor grantees’ activities at their locations
                     • Review expenditures of grantees compared with their approved
                       budgets and allowable grant expenditures, and review compliance
                       with grant requirements such as the submission of required grant
                       reportsa
    Close-out        •   Review and reconcile final data and other reports
                     •   Ensure that each recipient has met all of the grant’s programmatic
                         and financial requirements, returned any unused funds, and
                         provided final reports
Source: GAO
a
 For more on the grant monitoring and assessment functions of the three grant-making components
see GAO-12-517.




13
  In a competitive program, applicants generally compete for funding that agencies award
at their discretion. In contrast, in a formula program, recipients and award amounts are
generally based on statutorily defined calculations that may incorporate a state’s
population and violent crime rate, for example. For more details on these programs, as
well as cooperative agreements, see GAO-12-517.




Page 10                                                    GAO-13-92 DOJ Workforce Planning
DOJ and the Grant-Making   Office of Personnel Management (OPM) regulations require organizations
Components’ Workforce      to conduct workforce analyses as part of their human capital plans that
Planning Efforts           describe their current states, project human resources needed to achieve
                           organizational goals, and identify potential shortfalls. 14 In response, DOJ’s
                           JMD human resources staff has taken the lead in developing human
                           resource strategies and establishing a workforce planning framework for
                           the Department. In October 2007, JMD convened a workforce planning
                           council, comprised of representatives from the major DOJ components,
                           including OJP, to ensure that workforce planning and human resource
                           initiatives would be integrated consistently and cost-effectively throughout
                           DOJ. In June 2010, the council issued the Department’s first workforce
                           plan, covering the major components from 2010 through 2013. The plan
                           is updated each year using annual addenda. In addition to the workforce
                           plan, JMD issued a strategic leadership succession plan in 2007 and a
                           departmentwide human capital strategic plan covering 2007 through
                           2012, which it has recently begun to update. JMD also works with all the
                           grant-making components and other department components to refine
                           their budgets for submission in the President’s annual budget request.

                           Because OVW and the COPS Office are not considered to be major
                           components, and instead are referred to as “Offices, Boards and
                           Divisions,” JMD’s current workforce plans do not specifically address
                           these two grant-making components’ workload and workforce capacities
                           and gaps. In recognition of this, and in part because of congressional
                           interest in seeing more detail in the grant-making components’ budget
                           submissions, JMD took the lead in awarding a contract for a consolidated
                           workforce planning effort that would result in a staffing model for
                           assessing workload demands and workforce capacity—as well as gaps—
                           and strategies for bridging the gaps for each of the three grant-making
                           components. DOJ sent a request for proposals to select vendors in
                           September 2010, describing its requirement for a comprehensive
                           workload and workforce analysis to ensure that its grant-making
                           components are properly structured, staffed, and managed to meet
                           current and projected responsibilities. The request for proposals stated
                           that the review should identify best practices, problems, and solutions
                           relating to the grant-making components’ existing and projected workload
                           and workforce requirements, and should examine and document the
                           processes, procedures, and data used by each grant-making component



                           14
                            5 C.F.R. § 250.203.




                           Page 11                                        GAO-13-92 DOJ Workforce Planning
in making workforce planning decisions. DOJ awarded the contract in
early December 2010.

During the contract’s term, as well as during the months preceding the
final award, a number of events transpired that affected the grant-making
components’ plans for expanding the size of their workforces. As the
timeline in figure 2 illustrates, these events included a governmentwide
continuing resolution as well as a departmentwide hiring freeze. 15




15
  A continuing resolution is an appropriation act that provides budget authority for federal
agencies, specific activities, or both to continue in operation when Congress and the
President have not completed action on the regular appropriation acts by the beginning of
the fiscal year.




Page 12                                                 GAO-13-92 DOJ Workforce Planning
Figure 2: Timeline of Events That Affected the Grant-Making Components’ Plans to Expand the Size of their Workforces




                                        To meet DOJ’s specifications, the contractor provided regular progress
                                        reports, slide sets to explain the methods used, and a set of five key
                                        planning documents, or “deliverables,” as table 3 shows. In conducting
                                        this work, the contractor assigned staff to each of the grant-making
                                        components, and in May 2011, provided each with a unique set of these
                                        five deliverables, all tailored to the individual grant-making component’s
                                        circumstances and needs.




                                        Page 13                                           GAO-13-92 DOJ Workforce Planning
Table 3: Key Contractor Deliverables

 Deliverable                                       Purpose/scope
 Existing Workforce                                An initial profile of current and future workload (demand)
 Environment/Future                                and current and future workforce (supply).
 Workforce Environment
 Workforce Capacity Gap                            A report highlighting current and potential future talent
 Analysis                                          gaps at each skill and grade level within an occupational
                                                   series.
 A Description of the Staffing                     A description of the methods to develop the staffing
 Model and Related Talent                          model and strategies that each grant-making component
 Strategies                                        can execute to develop the workforce capacity and
                                                   competencies it needs for the future.
 Staffing Model                                    An electronic spreadsheet that reflects the work required
                                                   for key activities (e.g., grant administration) and matches
                                                   resources required with forecasted needs (e.g., newly
                                                   authorized grant programs). It was designed to be
                                                   flexible enough to react to future changes, to be used
                                                   and updated independently by grant-making component
                                                   staff, and to provide input into each grant-making
                                                   component’s workforce plans and resource/budget
                                                   requests.
 User Guide                                        A technical manual that documents the model’s
                                                   methodology and provides a tactical “how to” for using
                                                   the model. It was intended for audiences already familiar
                                                   with the work done at each grant-making component
                                                   and who will be actively using the model and its outputs
                                                   (e.g., senior leadership and other decision-makers,
                                                   model users, and model administrators).
Source: GAO analysis of contractor deliverables.



The staffing model is the key contractor deliverable that was designed to
provide DOJ’s grant-making components with:

1. a repeatable methodology for better estimating future resource needs
   and making fact-based cases for resources in their budget requests
   and

2. a tool for better (a) understanding the factors that drive their
   workloads, and (b) identifying the skill gaps they need to address to
   be successful in the future.

According to DOJ officials, the model was specifically designed as one
tool among many for workforce planning and resource allocations rather
than a single solution, and has the flexibility to evaluate the impact of
potential changes and scenarios.




Page 14                                                                   GAO-13-92 DOJ Workforce Planning
                       According to grant-making component officials, to inform the models’
                       development, they provided the contractor with access to subject matter
                       experts to inform the process through interviews, focus groups, and
                       surveys. A JMD official served as the lead Contracting Officer
                       Representative (COR) and each granting-making component designated
                       an assistant, or ACOR, to review the deliverables from the component’s
                       perspective. 16 The lead COR also chaired a technical evaluation panel,
                       comprised of a representative from each of the grant-making components,
                       to evaluate the contractor’s proposal prior to awarding the contract.


                       The grant-making components have partially met five of six leading
Grant-Making           practices, and fully met another, to ensure that their staffing models are
Components Have At     sound and reliable. We identified these six leading practices—broad
                       areas that represent a summary of the activities that should take place
Least Partially Met    during the three stages of model design, development, and deployment—
Leading Practices to   from prior GAO work. 17 They are: (1) accounting for risk associated with
                       contracting out model development, (2) ensuring the credibility of data
Ensure that the        used in the models, (3) preserving the integrity of data maintained in the
Staffing Models Are    models, (4) establishing roles and responsibilities for staff assigned to
Sound and Reliable,    update and operate the models, (5) ensuring adequate training for staff
                       assigned to update and operate the models, and (6) leveraging
But Can Strengthen     stakeholders’ knowledge and experience in designing and developing the
Their Efforts          models. We believe that fully adhering to these leading practices would
                       better position DOJ’s grant-making components to have reasonable
                       assurance that their staffing models are sound and reliable. Table 4
                       illustrates each of the six leading practices we included in our review,
                       their implications for soundness and reliability, and the extent to which the
                       grant-making components collectively met them.




                       16
                         The COR is responsible for ensuring that contractors abide by the contract. The COR
                       provides critical acquisition and technical functions to ensure that contractual agreements
                       are properly followed.
                       17
                         We primarily developed this framework using GAO-01-1008G. The leading practices are
                       also supported by other publications such as GAO-12-208G; GAO-09-232G; and
                       GAO-09-680G.




                       Page 15                                                 GAO-13-92 DOJ Workforce Planning
Table 4: Leading Practices for Ensuring Model Soundness and Reliability

                                                                                                                               Extent to which
                                                                                                                               components met the
Leading practice                                             Implications for soundness and reliability                        practice
Account for risk associated with contracting out             Weigh the benefits and hazards of contracting out for             Partially
model development                                            project management versus managing the project in-
                                                             house
Ensure the credibility of data used in the models            Provide assurance that data supporting a model,                   Partially
                                                             project, or activity are appropriate, complete, and
                                                             correct (e.g., check formulas in the model)
Preserve the integrity of data maintained in the             Provide assurance that changes to the model are                   Partially
models                                                       authorized, complete and accurate and are preserved
                                                             for ongoing use (e.g. password-protected data)
Establish roles and responsibilities for staff               Promote accountability and establish how resource                 Partially
assigned to update and operate the models                    needs will be met
Ensure adequate training for staff assigned to               Provide assurance that a model, activity, or project can          Partially
update and operate the models                                be used for its intended purpose
Leverage stakeholders’ knowledge and experience              Provide assurance that a model, project, or activity              Fully
in designing and developing the models                       meets user needs and specifications
                                                 Source: GAO analysis.

                                                 Note: A designation of “fully met” indicates that a component provided evidence that satisfied the
                                                 entire criteria for the respective leading practice; “partially met” indicates that a component provided
                                                 evidence that satisfied some portion of the criteria; and “did not meet” indicates that a component
                                                 provided no evidence that satisfied any portion of the criteria.


Account for Risk Associated                      The grant-making components partially met this leading practice, as table
with Contracting Out Model                       5 illustrates. Accounting for risk associated with contracting out model
Development                                      development includes verifying the contractor’s ability to complete the
                                                 work; outlining clear, relevant, and measurable objectives for the
                                                 contractor; and ensuring adequate technical assistance post-delivery of
                                                 the work product.

                                                 Table 5: Extent to Which Grant-Making Components Took Actions to Account for
                                                 Risk Associated with Contracting Out Model Development

                                                  Leading practice: account for risk associated with contracting out
                                                  model development                                                                             ◑
                                                   • Verify contractor's ability to complete work                                               ●
                                                   • Outline clear, relevant, and measurable objectives for the contractor                      ●
                                                   • Ensure adequate technical assistance post-delivery of work product                         
                                                      Not met

                                                 ◑     Partially met
                                                 ●     Fully met
                                                 Source: GAO.




                                                 Page 16                                                          GAO-13-92 DOJ Workforce Planning
To verify the contractor’s ability to complete the work, JMD and the grant-
making components confirmed the contractor had experience conducting
workforce and workload assessments. Specifically, the technical evaluation
panel requested information about the qualifications and experience of the
contractor staff who would be assigned to work on the models. Moreover,
to outline clear, relevant, and measurable objectives for the contractor, the
contract required the delivery of monthly contractor progress reports. JMD
appointed as lead COR a qualified individual with prior experience in
workforce analysis and ensured this individual received training in COR
roles and responsibilities. Together, the lead COR and the ACORs from
OJP, the COPS Office, and OVW reviewed these monthly progress reports
to check for completeness and timeliness of the work.

However, when deciding to contract out the work—rather than relying on
in-house staff—OJP, the COPS Office, and OVW stakeholders did not
design and execute the contract to ensure adequate technical assistance
post-delivery of the final work product. In particular, the components did
not address the provision of post-delivery technical assistance at any of
the following points during the contracting process:

•   when the grant-making components drafted an initial statement of
    work for JMD’s consideration;

•   when JMD prepared the final request for proposals to outsource the
    work;

•   when the technical evaluation panel reviewed the prospective
    contractor’s proposal—even though the proposal showed the work
    period ending with delivery of the models; and

•   when JMD approved the final contract.

As a result of not ensuring that the contract included a specific provision
for post-delivery technical assistance, all three grant-making components
have had to independently reach out to the contractor for help once they
started familiarizing themselves with the models and began to update
their data. As such, they have had to rely on the contractor’s goodwill to
answer any questions that have arisen since the contract period has
ended. For example, OVW officials stated that they were unable to
manipulate a formula in the staffing model to properly account for part-
time contractor staff. To work around this limitation, they had to enter
negative values elsewhere in the staffing model spreadsheet. OVW
officials reported reaching out to the contractor since they were not



Page 17                                        GAO-13-92 DOJ Workforce Planning
                                 knowledgeable about how to correct this issue on their own, and they
                                 ultimately received assistance in how to correct the error in the model.
                                 More recently, in August 2012, a COPS Office official reached out to the
                                 contractor for additional assistance updating their workforce data.
                                 Specifically, the COPS Office found that the model was not saving
                                 updates they had made. The contractor determined that there was a
                                 “bug” in the model, and proceeded to correct the issue. According to
                                 officials from all three grant-making components, at this time, they do not
                                 see a need for additional technical assistance; however, should it be
                                 needed, future assistance is not guaranteed and would require a new
                                 task order for future contracted services. 18

Ensure the Credibility of Data   The grant-making components partially met this leading practice, as table
Used in the Models               6 illustrates. Ensuring the credibility of data used in the models includes
                                 verifying with staff that the models accurately depict their system and
                                 testing the models for logic and accuracy.

                                 Table 6: Extent to Which Grant-Making Components Took Actions to Ensure the
                                 Credibility of Data Used in the Models

                                  Leading practice: ensure data credibility
                                      •   Verify with staff that model accurately depicts their system
                                                                                                                     ◑
                                                                                                                     ●
                                      •   Test the model for logic and accuracy
                                                                                                                     ◑
                                         Not met

                                 ◑        Partially met
                                 ●        Fully met
                                 Source: GAO.



                                 To verify with staff that the models accurately depict their respective
                                 systems, officials from the grant-making components told us that subject
                                 matter experts reviewed the accuracy of the staffing models’ workforce
                                 inputs, such as staffing levels, and workload assumptions, such as the
                                 length of time certain tasks take. For example, OJP officials stated that they
                                 reviewed the workforce data the contractor used from OPM to ensure that
                                 the full-time equivalent (FTE) staffing counts appeared reasonable.
                                 Moreover, OVW and COPS Office officials reported reviewing the
                                 contractor’s summaries of interviews with grants management staff to


                                 18
                                   A task order is an order for services placed against an established contract or with
                                 government sources. FAR § 2.101.




                                 Page 18                                                      GAO-13-92 DOJ Workforce Planning
determine the accuracy of the contractor’s estimates for completion of
certain grant management tasks, such as developing grant announcements
and notifying applicants of award decisions. Since these components do
not have systems to track the actual amount of time grant managers spend
per activity, they had no other way to verify the credibility of the workload
assumptions the contractor used to estimate resource needs.

With respect to testing the model for logic and accuracy, the components
reported taking steps to test the logic, but did not follow a systematic
process to test for accuracy in a manner consistent with leading practices.
To test the logic of the models’ design and development, the components
reported involving the ACORs, managers, and subject matter experts in
reviewing, evaluating, and approving the contractor’s assessments of
each component’s “existing environment.” OJP officials shared with the
contractor that they organize their workload around individual grant
solicitations rather than overarching grant programs. In contrast, officials
from the COPS Office and OVW ensured the contractor understood that
they organize their workload around the broader grant programs, rolling
up the individual solicitations. Thus, the contractor built the models in
recognition of these distinctions and the logic of each model reflects the
variation.

However, in testing the accuracy of the data and formulas in the model, the
grant-making components did not follow a systematic process to check the
completeness and accuracy of staffing model data. Specifically, in checking
the data, they did not use a random sample that was large enough to
conclude that all the data the contractor used to support the model were
sound and reliable. 19 Instead, all three grant-making components made
limited attempts to check the data and the formulas in the models they
received. For example, OJP officials stated they reviewed some of the data
but could not provide evidence of a sampling methodology that would allow
them to make conclusions about the overall accuracy of the data. Similarly,
officials from the COPS Office stated that they did not develop a systematic
verification plan to check that the data and formulas were free of flaws and
programming errors, but that they reviewed the model output and worked
backwards through the model to determine if the output seemed
questionable. OVW officials stated that they tested a sample of the data,
but did not use a scientific process to select that sample. They were unable



19
 GAO-01-1008G.




Page 19                                        GAO-13-92 DOJ Workforce Planning
to check all data because only one employee was available to perform this
task. According to component officials, the contractor’s experience with
similar projects in the private and public sectors provided them the
assurance that the contractor’s data gathering and modeling
methodologies were sound. Nevertheless, even with their limited review,
the grant-making components reported uncovering several contractor
errors. These ranged from the contractor’s formulas not carrying over
beyond a certain row in the spreadsheet—and thus not producing
calculations—to formulas not accounting for activities associated with grant
closeout activities—and thus underestimating total workload
responsibilities.

Our review of the models also identified several issues indicating that the
components did not test the accuracy of the models in a manner consistent
with leading practices for ensuring soundness and reliability. 20 For
example, the OJP, COPS Office, and OVW models each revealed a
programming error related to retirement calculations that none of the offices
was aware of before we identified it during our audit work. Specifically, the
OJP and COPS Office models were 1 year off in their retirement
projections, miscalculating the annual impact of potential retirement on the
existing workforce and, in so doing, miscalculating the potential capacity of
the workforce. Additionally, another methodological error in OJP’s model
caused OJP to initially underestimate retirements by 86 percent (or
approximately 54 retirements) from 2013 through 2016. OJP has since
corrected the errors in its model and the COPS Office is working to address
its error. Further, the methodological error in the OVW model causes any
staff eligible to retire in the next 12 months to be counted as eligible in the
current calendar year, irrespective of the actual year of eligibility. Though
OVW is a small office and the current error impacts just one employee, if
left uncorrected, this formula error could have greater impact in the future
as more employees approach retirement eligibility. For example, this error
could cause a model user to believe that the workforce would be impacted
on an accelerated timeframe and provide inaccurate information to inform
workforce planning. During the course of our audit work, OVW officials
could not identify the steps they planned to take to address the incorrect
formula. Additionally, the COPS Office’s model, unlike the other models,
omits eligible Executive Management staff from its retirement calculation,
resulting in an underestimation and suggesting that the overall rigor of the



20
 See appendix I for elaboration on this approach.




Page 20                                             GAO-13-92 DOJ Workforce Planning
                                 COPS Office review of the model may not have been adequate to ensure
                                 soundness. Upon acknowledgement of our finding, COPS Office officials
                                 said they planned to correct the error.

                                 When we asked the grant-making components about the extent of their
                                 initial tests of the models’ logic and accuracy, OJP officials stated they
                                 were satisfied with the retirement estimates that the model produced based
                                 on the contractor’s use of an assumed retirement rate of 15 percent per
                                 year. COPS Office officials stated that limited staff capacity—one person
                                 was assigned to the model—and the added challenge of reorganizing their
                                 Grants Administration Division while updating the model delayed more
                                 comprehensive checking of the contractor’s formulas. OVW officials also
                                 pointed to resource constraints, stating that they had sufficient resources to
                                 check only a sample and found this sampling approach to be reasonable in
                                 light of their limited staff capacity. Nevertheless, the errors we found raise
                                 questions about whether the components systematically confirmed the
                                 credibility of the staffing model data or verified the results. As a result, the
                                 reliability of the models could be compromised. Moreover, using the
                                 models without validating their results for decision-making could expose
                                 components to additional risk—for example, by miscalculating the size of
                                 the workforce needed to fulfill workload demands—ultimately impacting the
                                 efficiency and effectiveness of their operations.

Preserve the Integrity of Data   The grant-making components partially met this practice, as table 7
Maintained in the Models         illustrates. Preserving the integrity of the data maintained in the models
                                 includes four controls related to model use: authorization control;
                                 completeness control; accuracy control; and integrity control.

                                 Table 7: Extent to Which Grant-Making Components Took Actions to Preserve the
                                 Integrity of Data Maintained in the Models

                                  Leading practice: preserve data integrity
                                    •   Authorization control: authorize use with passwords, supervisory
                                                                                                                 ◑
                                        approval, and access rights                                              ●
                                    •   Completeness control: make changes electronically and review
                                        entries to ensure completeness                                           ●
                                    •
                                    •
                                        Accuracy control: check accuracy of data before and after entry
                                        Integrity control: protect model integrity by documenting changes
                                                                                                                 ◑
                                        and maintaining records                                                  ◑
                                       Not met

                                 ◑      Partially met
                                 ●      Fully met
                                 Source: GAO.




                                 Page 21                                                  GAO-13-92 DOJ Workforce Planning
With respect to authorization control, leading practices indicate that only
authorized users should be granted the ability to access and update the
staffing model. All three grant-making components have restricted access
to the models or plan to restrict access as a means of protecting their
results. OJP officials stated that access to the model is password-
protected to limit access to certain key individuals, and OVW officials
stated that they have provided copies only to those staff members
assigned to update and operate the model. Similarly, officials from the
COPS Office stated that access is restricted to the staff member assigned
to work on the model and the quality reviewer using their individual data
drives.

Regarding accuracy control, leading practices also suggest that users
should systematically review information for accuracy prior to entering
data and that others should check to identify and correct any errors
associated with data entry after data have been entered. Two of the three
grant-making components reported checking data prior to entry, and all
three grant-making components reported reviewing new data entries
against source documentation. For example, OJP officials stated that
data are checked post-entry, with two employees reviewing the data that
another employee has entered using the source documentation. COPS
Office officials reported a similar process in which data are first verified
with managers and then entered by one employee. Then, post-entry, all
data are reviewed by that employee’s supervisor using the same source
documentation. OVW officials reported checking personnel data from
JMD and contractor data from managers prior to entry. After data are
entered, a supervisor reviews the updates against the source documents,
and program-related data are double-checked by a program expert.

However, when asked to describe their processes for ensuring accuracy,
none of the components reported reconciling updated versions of the
model against prior versions—an action important for ensuring accuracy
after data are entered. Because each model is contained within a large
spreadsheet consisting of multiple worksheet tabs, reconciliation against
prior versions helps identify any changes made in error and ensure that
information from prior versions which should remain in the model is in fact
carried forward into any updated versions. Moreover, these stored
versions could allow the grant-making components to evaluate how
accurately the model predicted certain outcomes, such as attrition, and
ensure a record of workload and workforce changes over time.

With respect to integrity control, leading practices state that users should
document and maintain records of changes made to the model. Officials


Page 22                                        GAO-13-92 DOJ Workforce Planning
                             from all three grant-making components reported storing prior versions on
                             shared access servers or drives. Specifically, officials from OJP described
                             saving versions using a shared access server, which captures the most
                             recent version and prevents multiple users from updating the model
                             concurrently. Officials from the COPS Office and OVW reported saving
                             updated versions using the date of the update as a guide, but they did not
                             report using a process to ensure that the most recent version is used
                             each time data are updated and that multiple users may not concurrently
                             update the models. At the time of our review, only one employee from the
                             COPS Office and one employee from OVW were assigned primary
                             responsibility for updating the data, resulting in a low risk of updating the
                             wrong version or allowing concurrent updates. However, officials from
                             both the COPS Office and OVW reported that they intend to increase the
                             number of model users in the future, and implementing integrity controls
                             to identify the most current model version and prevent concurrent updates
                             would help protect model integrity.

                             As a result of not fully establishing comprehensive policies to review the
                             model for accuracy and documenting and maintaining records of changes
                             to protect model integrity, components increase the risk that the models
                             will not produce reliable information to inform management decisions and
                             budget requests. Both current decisions and future uses of the models
                             could be compromised if components do not act to implement protections
                             before using the models on a regular basis.

Establish Roles and          The grant-making components partially met this leading practice, as table
Responsibilities for Staff   8 illustrates. Establishing roles and responsibilities for staff assigned to
Assigned to Update and       update and operate the models includes assigning staff to all necessary
Operate the Models           tasks and defining their duties and responsibilities.

                             Table 8: Extent to Which Grant-Making Components Took Actions to Establish
                             Roles and Responsibilities for Staff Assigned to Update and Operate the Models

                              Leading practice: establish roles and responsibilities
                                                                                                        ◑
                                •   Assign staff to all necessary tasks
                                    Define the duties of responsible staff
                                                                                                        ◑
                                •                                                                       
                                   Not met

                             ◑      Partially met
                             ●      Fully met
                             Source: GAO.




                             Page 23                                             GAO-13-92 DOJ Workforce Planning
Before work began on model design, development, and deployment, all
three grant-making components assigned staff to some, but not all,
necessary tasks. Specifically, all three components identified and assigned
staff to work on the design and development of the models. Nevertheless,
delays in being able to use the model resulted in part because the grant-
making components did not identify staff to begin updating the models with
more recent data until many months after the contractor finalized the fiscal
year 2010 baseline data, as table 9 illustrates.

Table 9: Timeliness of Components’ Designation of Responsible Staff to Initiate
Updates to the Models

                                                                                                      Lapse in time
                           Designation of responsible staff to initiate                               from model
 Date                      updates to the models                                                      receipt
 January 2012              OVW reassigned an individual working under an                              8 months
                           existing contract to work with the model and update
                           its data.
 January 2012              OJP assigned a staff member it had hired in May                            8 months
                           2011 specifically to begin collecting and assessing
                           updated data.
 February 2012             COPS Office assigned a management analyst to                               9 months
                           start working on updates.
Source: GAO analysis of OVW, OJP and COPS Office documents and interviews with OVW, OJP and COPS Office officials.



OJP and OVW officials stated that delays in identifying staff responsible
for review and updates to the data in the models were due to staff
availability and timing of updates. OJP officials stated that they started
updates in January 2012 because that is when the individual assigned to
the model became available and when they had sufficient fiscal year 2012
data to include in the model. Similarly, OVW officials also stated that they
began the updates when 2012 data were available. Officials from the
COPS Office explained that competing human capital priorities and DOJ’s
hiring freeze contributed to delays in hiring and assimilating qualified staff
with the skills needed to work on the model. All three components have
faced or currently face human resource constraints and challenges in
hiring and staffing qualified staff with the skills needed to update data in
the staffing models. For example, OJP was authorized for more human
capital staff than employed until October 2012, which is when it expected
to fill its three open human capital positions, and one of the individuals
filling these three slots would be assigned to work on the model. OVW
officials stated that they are authorized for one human capital position, but
the slot remains vacant due to budget concerns and the associated
departmentwide hiring freeze that remains in place. According to OVW



Page 24                                                                     GAO-13-92 DOJ Workforce Planning
officials, while JMD fulfills most of their human capital needs, having this
position vacant means they have no one to focus on strategic human
capital planning within their organization. Though OVW officials said they
could seek a hiring exemption to fill this slot, they have chosen instead to
request exemptions to fill more critical vacancies associated with grants
management. 21 Officials from the COPS Office stated they have only one
human capital position authorized and filled, and have had to “borrow”
staff from other units in order to complete their model updates. Because
the three grant-making components already faced constrained staffing
and limited personnel resources prior to accepting the additional
responsibility of operating the models, it was even more imperative that in
preparation for model deployment, they identified the human capital
capacity needed to carry out the related staffing model responsibilities.

Moreover, the grant-making components did not define the duties of staff
responsible for working with the model before staff members were
brought onboard. Officials from all three grant-making components stated
that newly assigned staff were not part of the discussions with the
contractor on the model’s design and development, and required
additional time to familiarize themselves with the model’s data before they
could begin making updates. For example, the individual OVW tasked
with model updates said that, initially, nearly 75 percent of her time was
spent familiarizing herself with the model in order to get up to speed on
the model’s capabilities. Although all three grant-making components
currently have staff onboard to continue updates to the data in the
models, it is unclear if these assignments can be sustained and thus, if
the roles and responsibilities can be institutionalized. For example, COPS
Office officials stated that the one COPS Office staff member originally
assigned responsibility for managing the model left the Department. Her
successor, an accountant within the COPS Office, has been assigned to
work with the model as time permits, given her other officially assigned
workload and responsibilities. Similarly, the staff member OVW assigned
with responsibility for data updates is a contractor and OVW is not certain
about renewal of the company’s contract. Going forward, clearly


21
  Each grant-making component decides how many hiring exemptions to seek and then
sends these requests to JMD for review and recommendation. JMD reviews each
exemption for a number of elements, which, among others, includes the position
description, an explanation for why current staff are unable to fulfill the duties and
responsibilities of the position, and the availability of funding to support the position in the
future. If JMD recommends approval, JMD forwards the hiring exemption request to the
Deputy Attorney General for a final decision.




Page 25                                                    GAO-13-92 DOJ Workforce Planning
                               delineating roles and responsibilities would help DOJ institutionalize the
                               processes required to manage the staffing models and would enhance
                               DOJ’s grant-making components’ ability to sustain, and, more importantly,
                               facilitate workforce planning and budget development decision-making.

Ensure Adequate Training for   The grant-making components partially met this practice, as table 10
Staff Assigned to Update and   illustrates. Ensuring adequate training for staff assigned to update and
Operate the Models             operate the models includes ensuring a mechanism exists for providing
                               training to staff and ensuring that training is comprehensive and sufficient
                               to enable successful operation of the model.

                               Table 10: Extent to Which Grant-Making Components Took Actions to Ensure
                               Adequate Training for Staff Assigned to Update and Operate the Models

                                Leading practice: ensure adequate training
                                  •   Ensure mechanism exists for providing training to staff
                                                                                                               ◑
                                                                                                               ●
                                  •   Ensure training is comprehensive and sufficient to enable successful
                                      operation of the model                                                   
                                     Not met

                               ◑      Partially met
                               ●      Fully met
                               Source: GAO.



                               JMD ensured a mechanism existed for providing staff training by stating
                               in the contract that the contractor would be required to provide training
                               sufficient to allow grant-making component staff to operate the staffing
                               model and update it to enable ongoing use. To meet this requirement and
                               with JMD’s concurrence, the contractor delivered User Guides to each
                               grant-making component, which were designed to serve as a training
                               mechanism for staff who would be updating and using the models.

                               However, the components did not ensure that the guides were
                               comprehensive and sufficient to enable successful operation of the
                               models. Specifically, they did not thoroughly examine the content of the
                               guides upon receipt, but rather waited to review them carefully until they
                               began to update the models. Moreover, component officials told us of
                               their varying levels of satisfaction with the User Guides—one component
                               indicated that the guide was helpful; a second component reported that it
                               was useful, but noted several functions that the Guide should have
                               included or better explained; and the third component stated that the
                               guide did not meet its needs. When we asked the grant-making
                               component officials about training beyond the guides, OJP and COPS



                               Page 26                                                  GAO-13-92 DOJ Workforce Planning
                               Office officials said the contractor did not provide any, while the OVW
                               official who currently reviews the model’s updates said she recalled some
                               hands-on training from the contractor to facilitate her use. Since none of
                               the grant-making components’ primary model users were in their current
                               positions when the contractor was competing its work, all reported having
                               to develop familiarity with the models via hands-on, on-the-job exposure.

                               As a result of not receiving training, components have experienced delays
                               in updating the staffing model to enable its use for budget formulation and
                               workforce planning. Model users from OJP, the COPS Office, and OVW
                               reported that the delays in updating the models resulted, in part, from
                               needing additional time to familiarize themselves with the model. As users
                               have gained more familiarity with the models, some of the components
                               have made efforts to train other users, which is consistent with the
                               contractor’s notes in one of its planning documents, which states that in a
                               hiring freeze environment, capacity gaps will have to be addressed
                               through other means, such as cross training. For example, OVW officials
                               stated that the individual currently assigned primary responsibility for
                               updating the model began to cross-train another OVW human resources
                               staff member. As stated previously, the COPS Office recently lost its
                               original model user due to attrition, and as of August 2012, has utilized a
                               staff accountant whose workload contained additional capacity in order to
                               proceed with model updates. Like the COPS Office, OJP officials have
                               just one person assigned to the model and no back-fill available.

Leverage Stakeholders’         The grant-making components fully met this leading practice, as table 11
Knowledge and Experience in    illustrates. Leveraging stakeholders’ knowledge and experience in
Designing and Developing the   designing and developing the models includes involving stakeholders in
Models                         establishing contractor terms of work and involving stakeholders in design
                               and development.

                               Table 11: Extent to Which Grant-Making Components Took Actions to Leverage
                               Stakeholders’ Knowledge and Experience in Designing and Developing the Models

                                Leading practice: leverage stakeholders                                         ●
                                     Involve stakeholders in establishing contractor terms of work             ●
                                     Involve stakeholders in design and development                            ●
                                     Not met

                               ◑      Partially met
                               ●      Fully met
                               Source: GAO.




                               Page 27                                                   GAO-13-92 DOJ Workforce Planning
Stakeholder involvement was evident in establishing the contractor terms
of work. According to OJP officials, OJP produced the initial draft
statement of work to outline the scope and terms of the prospective
contractor’s work, and stakeholders from the COPS Office and OVW
provided their input by commenting on the draft. Further, OJP, the COPS
Office, and OVW each appointed an ACOR to serve as a stakeholder on
the technical evaluation panel, which was tasked with evaluating the
contractor’s proposal. As part of its proposal selection process, this
technical evaluation panel submitted two sets of follow-up questions to
the prospective contractor seeking additional information and clarifying
aspects of the proposal. For example, the panel desired clarification,
which they eventually received, on how the contractor would compare the
grant-making components’ operations against other public and private
organizations. The panel then evaluated the prospective contractor’s
responses in conjunction with the overall proposal before DOJ finalized
the contract. From the COR’s perspective, leveraging the perspectives of
stakeholders from the grant-making components—via this panel—helped
ensure that the stakeholders were engaged in the contractor selection
process and ultimately in agreement regarding the services the contractor
aimed to provide.

Stakeholder involvement also was evident during the models’ design and
development. As noted, DOJ’s initial request for proposal required the
contractor to submit monthly progress reports, and the lead COR
reviewed these to check for completeness and timeliness of the work.
According to officials from OJP, the COPS Office, and OVW, the lead
COR reached out to the ACORs to confirm with these stakeholders that
the contractor’s work met their needs. For example, officials from the
COPS Office and OJP stated that when the contractor concluded its
assessments of the conditions at the COPS Office and OJP at the start of
the work period, the COR asked each component’s ACOR to accept or
decline the assessment. Similarly, OVW officials stated that the COR met
with OVW’s ACOR to discuss the accuracy of the contractor’s
assessment of the conditions at OVW at the start of the work period. All
three of the grant-making components reported that their respective
senior management—yet another set of stakeholders—was involved in
confirming the accuracy of the contractor’s assessments of each
component’s existing conditions.

Moving forward, fully incorporating leading practices when updating the
models for future use will better position the grant-making components to
capitalize on the models’ capacity and facilitate effective workforce
planning and budgeting. Specifically, implementing a more rigorous data


Page 28                                      GAO-13-92 DOJ Workforce Planning
                             review process with clear methods for confirming the credibility of data
                             and verifying the results the staffing models generate; reviewing changes
                             to the data in the models to ensure accuracy; defining and determining
                             the respective roles and responsibilities of staff involved in reviewing,
                             operating, and assisting with the staffing models; and identifying training
                             needs that address staff needs to fulfill responsibilities associated with the
                             staffing models would better position the grant-making components to
                             facilitate effective workforce planning and make sound budget
                             development decisions.


                             Due to the deficiencies we identified in the design, development, and
Grant-Making                 deployment of the staffing models, the grant-making components have
Components Have              not used the staffing models to determine workforce gaps, such as critical
                             skills and competencies of staff, or to develop long-term human capital
Not Used the Staffing        plans that address these gaps. Additionally, the three grant-making
Models to Determine          components have not yet begun to use the data in the models to inform
                             the preparation of annual budget requests. However, officials from all
Workforce Gaps and           three grant-making components stated that they would like to or plan to
Do Not Have a                use the models to inform the budget submissions for fiscal year 2015
                             funding, but they do not yet have a strategy in place for doing so.
Strategy for Using the
Models to Inform
Budget Requests

The Grant-Making             Our prior work examining issues in strategic human capital planning
Components Have Not          highlighted the use of workforce planning tools, such as staffing models,
Used the Staffing Models     to assist organizations in addressing two critical needs in strategic
                             workforce planning: (1) aligning an organization’s human capital program
to Identify Capacity Gaps,   with its current and emerging mission and programmatic goals, and (2)
but Some Have Used the       developing long-term strategies for acquiring, developing, and retaining
Models to Support            staff to achieve programmatic goals. The three grant-making components
Ongoing or Operational       have not yet begun to use the models in either of these two areas, in part,
                             because of some of the staffing challenges noted earlier. However,
Planning                     according to officials from all three grant-making components, they are
                             pursuing various staffing options, such as exemptions to the
                             departmentwide hiring freeze to dedicate staff to work on the models,
                             reorganizing existing staff and shifting their responsibilities, or using
                             contractor personnel, so that they can expand their use of the model
                             accordingly.




                             Page 29                                        GAO-13-92 DOJ Workforce Planning
As of September 2012, OJP received approval for three hiring
exemptions intended to enhance the component’s human capital
capacity. According to OJP officials, once they fill these positions—
expected by October 2012—the current, sole human resources specialist
assigned to update the model will have support from one additional staff
member. Officials from the COPS Office and OVW stated that they are
not seeking hiring exemptions to add to their human capital capacity.
However, according to officials from the COPS Office, they are discussing
the possibility of moving internal staff from other offices to address their
human capital needs. OVW officials stated that they are continuing to
utilize a contracted staff person to work on updates to the model and that
any hiring exemptions OVW submits will concentrate on positions that
directly relate to its grant-making responsibilities.

Nevertheless, OJP officials stated that they used the staffing model’s data
to help determine who among their staff would be eligible for 2012
Voluntary Early Retirement Authority (VERA) and Voluntary Separation
Incentives Payment (VSIP)—two OPM initiatives that help components
resize their workforces. In addition, OJP officials explained that when two
OJP program offices separately considered whether to seek exemption
authority to hire more staff, OJP’s Human Resources Division relied on
the model to provide data on the number and type of positions each office
already had, and served as one tool to show that consolidation across the
two offices was plausible. As a result, OJP is currently considering
consolidation of these offices rather than expanding one or both of them.
At a more conceptual level, OVW officials said that OVW used the
model’s data to confirm a previously held workforce planning assumption
that the workload was increasing as the workforce was decreasing (due
to attrition and lag time in hiring replacement staff). As such, officials
reported that they envision the model being useful to them as they
embark on decisions about whether to reorganize OVW. In particular,
they said they plan to use the model to show the areas in the grant life
cycle in which staff members spend most of their time and what
processes OVW can improve to achieve greater efficiency. In addition,
the COPS Office’s restructuring is underway, and now that the model
accurately accounts for their new workload and assignment of
responsibilities, officials plan to analyze the model’s results for ongoing
strategic workforce allocations.




Page 30                                       GAO-13-92 DOJ Workforce Planning
The Components Do Not        The grant-making components reported that they would like to or plan to
Yet Have a Documented        use the models to inform the fiscal year 2015 budget request; however,
Strategy to Ensure the       as of September 2012, they did not yet have documented plans in place
                             to do this, even though the budget planning process will begin in early
Staffing Models Inform the   2013 and according to OJP officials can start as early as late fall 2012.
Fiscal Year 2015 Budget
Request                      As noted, the contractor delivered the staffing model in May 2011 based
                             on fiscal year 2010 workload and workforce data, which was the most
                             recently available at the time. According to grant-making component
                             officials, they were unable to use the models to inform their budget
                             development decisions in the fall of 2011 (to inform the request for
                             funding in fiscal year 2013) because they considered the staffing models’
                             “snapshot” data already too old—that is, not reflective of current
                             conditions—for reasonable inclusion. Thus, the grant-making components
                             spent the first year after the contractor delivered the model learning its
                             functionality and ramping up staff and expertise to update it. The grant-
                             making components have completed these updates as of September
                             2012, and officials from the three grant-making components reported that
                             they would like to or plan to use the staffing models to inform the
                             upcoming budget request for fiscal year 2015. Consequently, the grant-
                             making components could better position themselves to do so by
                             beginning the planning process early. In this regard, standard practices
                             for project management state that managing a project involves, among
                             other things:

                             •    developing a timeline with milestone dates, including specific points
                                  throughout the project to reassess efforts underway and determine
                                  whether changes are necessary;

                             •    establishing coordination to understand the appropriate roles of those
                                  involved with a project and how they engage the various stakeholder
                                  groups; and

                             •    providing guidance regarding the direction of a project and changes
                                  affecting the outcome of projects. 22




                             22
                               Project Management Institute, The Standard for Program Management® (Newtown
                             Square, PA: 2006). The Project Management Institute is an organization that provides
                             guidelines, rules, and characteristics for project management.




                             Page 31                                               GAO-13-92 DOJ Workforce Planning
            Implementing a documented project management strategy to include
            timelines, coordination efforts, and guidance for using the data in the
            staffing models to inform the fiscal year 2015 budget submission would
            help the grant-making components leverage the quantifiable workforce
            and workload data that the models generate and provide context and
            justification to the Congress for the funding levels they request.

Timelines   According to grant-making component officials, working with the data in
            the staffing models necessitates time to validate and verify data going into
            the models, update the models, and review updates to the models for
            completeness and accuracy. However, officials from the grant-making
            components provided no specific timeframes to address a comprehensive
            analysis of the data, which could ensure the full incorporation of leading
            practices, as we have discussed. Rather, officials from the COPS Office
            and OVW stated that they each anticipate twice-a-year updates to the
            data in the staffing models: (1) when the President’s Budget is prepared,
            and (2) when each component receives its actual appropriation. Officials
            from OJP stated that they also anticipate twice-a-year updates to the data
            in the staffing model, but OJP officials are planning to update the OJP
            model in January of each year and then again at the end of each fiscal
            year. Nevertheless, during the course of our audit, the updates each
            grant-making component had been making to the model have been
            limited to ensuring the data are most current instead of comprehensively
            analyzing the data and determining what the analysis means for
            component operations. Prospectively, officials from all three grant-making
            components stated that it is their goal to conduct a comprehensive
            analysis of data in future updates that will help them improve budget
            planning.

            Grant-making component officials also explained that time will be
            necessary for those working with the model to coordinate with those
            developing the budget; however, they also have not established timelines
            for such interaction. According to OJP officials, they expect that the
            model’s 2012 data will inform the fiscal year 2015 budget request and
            acknowledge that OJP’s human capital staff will need to share and
            interpret the data in the models with OJP’s budget staff. To this end, OJP
            officials reported in September 2012 that both human capital and budget
            staff have recently held planning meetings but that more time is
            necessary to finalize their plans. Officials from the COPS Office and OVW
            stated that they have not yet set aside time to conduct similar planning
            meetings. According to officials in the COPS office, they brought on a
            new staff member in August 2012 to work on updates, and she will need
            to rely on additional data from other COPS program offices to update the


            Page 32                                       GAO-13-92 DOJ Workforce Planning
               model with the most current data before informing the fiscal year 2015
               budget request. OVW officials stated that staff constraints have limited
               their ability to start planning meetings between human capital and budget
               staff. The officials emphasized that OVW is a small office, and
               specifically, within the budget and grants management units, staff
               members have been focused on closing out grants for the 2012 fiscal
               year rather than thinking ahead to the 2015 budget cycle. They hope to
               start holding initial meetings by November or December 2012 but these
               plans are tentative. While the offices have begun to discuss the
               importance of timing, without establishing specific timeframes that take
               into account the full incorporation of leading practices, as well as the
               exchange of information between those running the models and those
               developing the budgets, components risk missing an opportunity to
               ensure that the models are sound and reliable and that what they are
               learning is supporting what they are requesting in their budgets.

Coordination   Staff members from the grant-making components stated they did not yet
               know how they would use the results that the staffing models generated
               to develop and support budget submissions. For example, at this time,
               human capital and budget offices from all three grant-making components
               have demonstrated limited efforts to coordinate their roles and identify
               what information human capital staff will provide to budget staff and what
               the budget office staff responsibilities will be to validate any of the
               analysis the models’ users generate. Moreover, coordination between
               human capital and budget office staff needs to take place not only to
               inform the budget requests, but also to understand the staffing model
               data and the analyses model users are conducting. OJP officials stated
               that human capital and budget staff members have met to discuss
               coordination efforts to prepare OJP’s budget request for fiscal year 2015
               and that they continue to outline roles and responsibilities. Additionally,
               OJP’s Acting Assistant Attorney General issued a memorandum to OJP
               leadership in October 2012 that directed OJP offices to take the staffing
               model into consideration when developing budget requests. The memo
               did not provide specific instructions for how to coordinate, but did
               emphasize the importance of integrating workforce, program, and budget
               planning. Officials from the COPS Office agreed that coordination
               between human capital and budget offices should occur. They reported
               general discussions between the COPS Office Chief of Staff and the
               COPS Office budget unit on how to integrate the model into budget
               development, but acknowledged that they have not discussed specifics
               yet and said they planned to begin formal coordination activities in the
               spring of 2013, which is when DOJ components have typically begun
               preparing for an upcoming budget cycle. OVW officials stated that initial


               Page 33                                      GAO-13-92 DOJ Workforce Planning
           coordination steps are necessary for budget staff to understand the
           intricacies of the model in order to eventually decide how and what data
           in the model will be most useful to justifying the budget. However, OVW
           officials reported that limited staff and growing demands on their time
           have limited their ability to begin coordination efforts between human
           capital and budget staff.

           As sound project management approaches suggest, involving planning
           and operational personnel early is meant to avoid making decisions in
           isolation that may waste resources and time. The future coordination
           efforts between human capital and budget office staff should help
           determine how the staffing models can best be used to justify any
           requests for new funds. Mechanisms to facilitate coordination and
           collaboration could enhance DOJ grant-making components’ budget
           submissions by making them more consistent, accurate, and robust.
           Further, because development of the fiscal year 2015 budget submission
           will reflect the components’ first attempt to integrate analyses from the
           staffing models, it is increasingly important that conversations between
           human capital and budget staff begin early and be well-coordinated.

Guidance   Staff members at each grant-making component have become more
           familiar with the staffing models and their capabilities, but to date, there
           has been no guidance about what analyses may be needed or useful
           when developing budget requests. According to JMD officials, they do not
           intend to dictate how and to what extent the components incorporate the
           models into their budget submissions. JMD officials told us that during
           their routine review of the budget justifications the grant-making
           components submit, they may question the level of detail included or
           omitted, but they do not envision their role as ensuring inclusion or even
           securing consistency across the three grant-making components’
           requests. From the perspective of JMD’s budget staff, JMD has never
           inserted itself into DOJ components’ budget development processes and
           has no directive from the Attorney General to do so. Further, JMD does
           not plan to modify its approach given the presence of the staffing models.
           While guidance may not be necessary from JMD, guidance from each
           grant-making component outlining the process for developing budget
           submissions through incorporation of the model’s data could help ensure
           that on a routine basis, updates to the model incorporate leading
           practices and that budget development discussions are based on sound
           and reliable data and associated workforce and workload analyses.

           Developing, documenting, and implementing a strategy with timelines,
           coordination mechanisms between budget and human capital staff, and


           Page 34                                       GAO-13-92 DOJ Workforce Planning
              guidance on how to incorporate the models’ into budget development
              could better position grant-making components to use the quantifiable
              data the models provide to support the funds they seek in their budget
              requests. Having such a strategy could also provide managers at the
              grant-making components and other decision makers insights into the
              grant-making components’ overall progress in using the models and
              provide a basis for determining what, if any, additional resources are
              needed to best manage their operations.


              During fiscal year 2011, the workforce at OJP, the COPS Office, and
Conclusions   OVW collectively administered nearly 27,000 active grants involving $3.5
              billion in appropriated funds. Given the volume and overall size of these
              grants, it is critical that OJP, the COPS Office, and OVW use their
              resources in the most efficient and effective way possible. Since 2010,
              DOJ has embarked on an effort to quantify the grant-making components’
              workloads, assess their workforces, and identify and address any gaps by
              funding the development of staffing models. As a result, DOJ’s grant-
              making components now have increased capacity to engage in
              systematic and strategic workforce planning.

              The grant-making components have partially met leading practices to
              ensure the models are sound and reliable, but can enhance their efforts
              to more fully meet leading practices. For example, not fully checking for
              data accuracy upon the model’s receipt and attempting to correct for
              errors after the contractor’s term had ended challenge the components’
              abilities to put the models to their intended use—to determine workforce
              gaps, such as critical skills and competencies of staff, and to develop
              long-term human capital plans that address these gaps. Fully meeting
              leading practices would provide DOJ reasonable assurance that, among
              other things, the models’ data are accurate, the results are valid, and the
              staffing models provide a sound and reliable basis for components to
              update their models, analyze the data in the future, and inform workforce
              and budget planning. Further, concentrating for so many months on
              updating the models’ baseline data has left the components without a
              strategy for analyzing the models’ results and applying what they learn to
              the development of their budget requests. By developing, documenting,
              and implementing a strategy that includes timelines, coordination
              principles, and guidance for incorporating the staffing models’ analyses
              into their budget development processes, the components may be better
              positioned to leverage the quantifiable workforce and workload data the
              models generate, inform the Congress of their needs, and more fully
              justify any requests for additional funds in the future.


              Page 35                                       GAO-13-92 DOJ Workforce Planning
                      We recommend that the Attorney General direct the leadership of OJP,
Recommendations for   the COPS Office, and OVW take the following two actions.
Executive Action
                      •   fully incorporate leading practices to ensure that their respective
                          staffing models are sound and reliable, by:

                          •     implementing a more rigorous data review process with clear
                                methods for confirming the credibility of data and verifying the
                                results the staffing models generate;

                          •     reviewing changes to the data in the models to ensure accuracy
                                and documenting and maintaining records of changes to protect
                                model integrity;

                          •     defining and determining the respective roles and responsibilities
                                of staff involved in reviewing, operating, and assisting with the
                                staffing models; and

                          •     identifying training needs that address staff needs to fulfill
                                responsibilities associated with the staffing models.

                      •   develop, document, and implement a strategy that includes timelines,
                          coordination efforts, and guidance for building budget submissions
                          that takes into account the full incorporation of leading practices to
                          ensure the staffing models are sound and reliable.


                      We provided a draft of this report to DOJ for comment and received
Agency Comments       written comments on the draft report, which are reproduced in full in
                      appendix II. DOJ concurred with both recommendations and described
                      actions it has underway or plans to take to address the recommendations.

                      DOJ agreed with the first recommendation that the leadership of OJP, the
                      COPS Office and OVW fully incorporate leading practices to ensure that
                      their respective staffing models are sound and reliable, noting that it
                      appreciated our evaluation of the staffing models. DOJ stated that with
                      available resources, the grant-making components plan to continue to
                      integrate workforce planning into their operational, programmatic, and
                      budget planning processes, and expressed their commitment to a long-
                      term, systematic, and data-driven approach to doing so. Additionally, DOJ
                      reported that the grant-making components are currently benchmarking
                      against federal agencies and the private sector to ensure that their




                      Page 36                                           GAO-13-92 DOJ Workforce Planning
workforce and talent management efforts are developed and/or enhanced
based on best practices and industry standards.

DOJ agreed with the second recommendation to develop, document, and
implement a strategy that includes timelines, coordination efforts, and
guidance for building budget submissions that takes into account the full
incorporation of leading practices to ensure the staffing models are sound
and reliable. DOJ noted that staff in the human capital, program, and
budget offices of the grant-making components will continue to
collaborate on a strategic approach to the development of future budget
submissions and work to establish policies and procedures to govern this
coordinated effort.


We are sending copies of this report to the Attorney General, selected
congressional committees, and other interested parties. In addition, the
report is available at no charge on the GAO website at
http://www.gao.gov.

If you or your staff have any questions about this report, please contact
me at (202) 512-9627 or maurerd@gao.gov. Contact points for our
Offices of Congressional Relations and Public Affairs may be found on
the last page of this report. Key contributors to this report are listed in
appendix III.




David C. Maurer
Director
Homeland Security and Justice Issues




Page 37                                        GAO-13-92 DOJ Workforce Planning
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              This report addresses the following questions: (1) to what extent have the
              grant-making components met leading practices to ensure that their
              staffing models are sound and reliable?, and (2) how have the
              components used the models to inform their workforce planning efforts
              and to what extent have the components begun to use the models for
              budget development?

              To determine the extent to which the three grant-making components
              incorporated leading practices to ensure that their staffing models are
              sound and reliable, we developed a detailed framework to define six
              leading practices and 15 specific activities, or criteria, they encompass.
              The framework covers model design, model development, and model
              deployment—and the six leading practices reflect the specific activities in
              which organizations should engage to ensure the models’ soundness
              (i.e., their validity and reliability). These practices are: (1) accounting for
              risk associated with contracting out model development, (2) ensuring the
              credibility of data used in the models, (3) preserving the integrity of data
              maintained in the models, (4) establishing roles and responsibilities for
              staff assigned to update and operate the models, (5) ensuring adequate
              training for staff assigned to update and operate the models, and (6)
              leveraging stakeholders’ knowledge and experience in designing and
              developing the models.

              We developed this framework primarily using GAO’s Internal Control
              Management and Evaluation Tool. 1 The leading practices are also
              supported by GAO’s Designing Evaluations: 2012 Revision; Federal
              Information System Controls Audit Manual; and Assessing the Reliability
              of Computer-Processed Data. 2 Before applying the framework, and to
              understand how the staffing models are structured, we participated in a
              demonstration of the models in which each grant-making component
              presented its respective version of the model. During this demonstration,
              officials responsible for operating the model at their respective
              components walked us through the various model functions, identified
              their underlying data sources, and explained the model outputs they
              produced. We also independently reviewed the models and the
              associated deliverables the contractor provided. Finally, we interviewed
              grant-making component officials assigned to operate the model to


              1
                  GAO-01-1008G.
              2
               GAO-12-208G; GAO-09-232G; GAO-09-680G.




              Page 38                                          GAO-13-92 DOJ Workforce Planning
Appendix I: Objectives, Scope, and
Methodology




confirm our understanding of the models’ structure. Then, to apply the
framework, we assessed the steps the grant-making components told us
they took to ensure the models were sound and reliable and
benchmarked those steps against the leading practices and criteria we
identified. A designation of “fully met” indicates that a component
provided evidence that satisfied the entire criteria for the respective
leading practice; “partially met” indicates that a component provided
evidence that satisfied some portion of the criteria; and “did not meet”
indicates that a component provided no evidence that satisfied any
portion of the criteria.

For example, to understand how the Department’s Justice Management
Division (JMD) and its three grant-making components worked with the
contractor to design and develop the model, we interviewed the lead
Contracting Officer Representative (COR)—a management analyst within
JMD—as well as senior officials from the three grant-making components
responsible for grant administration, human capital management, and
budget planning. Additionally, we interviewed these officials to understand
how the grant-making components have used the models for workforce
planning and budget development efforts since receiving the models from
the contractor. We also interviewed the staff in each grant-making
component tasked with updating and executing the model to get their
perspectives on the training they received, their prior workforce planning
backgrounds, and the time they spent working with the model. In addition,
we analyzed documentation about the grant-making components’
practices related to updating the models, as well as the contractor’s
guidelines for doing so. We then compared the grant-making
components’ practices for designing, developing, and deploying the
models for eventual use to the leading practices we identified and then
assessed whether the grant-making components’ actions fully met,
partially met, or did not meet the criteria. We also assessed each grant-
making component’s model independently, using a discovery method
approach, wherein we examined a small number of formulas and
calculations. While we were not attempting to develop a specific error rate
per se, this discovery method approach permitted us to make conclusions
about the risk for (a) additional errors in other parts of the model we did
not examine, and (b) potential errors in the model’s outputs.

To describe how the three grant-making components have used the
models to inform their workforce planning efforts and to determine to what
extent to which they have begun to use the models for budget
development, we interviewed officials from the three grant-making
components and JMD officials on their workforce planning efforts prior to


Page 39                                      GAO-13-92 DOJ Workforce Planning
Appendix I: Objectives, Scope, and
Methodology




having the staffing models. In particular, we interviewed staff members
from each of the three grant-making components who were working in a
human capital capacity and had direct oversight or responsibility for the
staffing models. In these interviews, we discussed the extent, if any, to
which each grant-making component was using the model to inform its
activities, future plans to integrate the model into its budget development
process, and the role JMD plays in facilitating these efforts. In addition,
we reviewed related documents including each grant-making
component’s initial staffing models provided by the contractor in May
2011 and the updated models completed by each grant-making
component as of September 2012. In reviewing the three grant-making
components’ plans to integrate the models into their budget development
process, we relied on standard practices for project management, The
Standard for Project Management, developed by the Project Management
Institute. 3 Through interviews with component officials at the three grant-
making components and review of documentation including meeting
notes, we reviewed the grant-making components’ use of the standard
practices for project management including the development of timelines
with milestone dates, coordination efforts between human capital and
budget staff at each grant-making component, and guidance regarding
the direction and use of the models.




3
 Project Management Institute, The Standard for Program Management® (Newtown
Square, PA: 2006). The Project Management Institute is an organization that provides
guidelines, rules, and characteristics for project management.




Page 40                                               GAO-13-92 DOJ Workforce Planning
Appendix II: Comments from the Department
             Appendix II: Comments from the Department
             of Justice



of Justice




             Page 41                                     GAO-13-92 DOJ Workforce Planning
Appendix II: Comments from the Department
of Justice




Page 42                                     GAO-13-92 DOJ Workforce Planning
Appendix II: Comments from the Department
of Justice




Page 43                                     GAO-13-92 DOJ Workforce Planning
Appendix II: Comments from the Department
of Justice




Page 44                                     GAO-13-92 DOJ Workforce Planning
Appendix III: GAO Contact and Staff
                  Appendix III: GAO Contact and Staff
                  Acknowledgments



Acknowledgments

                  David C. Maurer, (202) 512-9627 or maurerd@gao.gov
GAO Contact
                  In addition to the contact named above, key contributors to this report
Staff             were Joy Booth (Assistant Director), Lydia Araya, Sylvia Bascopé,
Acknowledgments   Christine Hanson, William Mowbray, Daniel Rodriguez, Rebecca Shea,
                  and Janet Temko.




(441079)
                  Page 45                                      GAO-13-92 DOJ Workforce Planning
                      The Government Accountability Office, the audit, evaluation, and
GAO’s Mission         investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (http://www.gao.gov). Each weekday
GAO Reports and       afternoon, GAO posts on its website newly released reports, testimony,
                      and correspondence. To have GAO e-mail you a list of newly posted
Testimony             products, go to http://www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
                      Website: http://www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in   E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.