Report 2013-003-CARO - Cooperative Audit Resolution and Oversight Initiative (CAROI) Pilot Program

Published by the Equal Employment Opportunity Commission, Office of Inspector General on 2014-09-23.

Below is a raw (and likely hideous) rendition of the original report. (PDF)




    United States Equal Employment Opportunity Commission
                   Office of Inspector General
      Cooperative Audit Resolution and Oversight Initiative
                          Pilot Program

                        Results in Brief

September 23, 2014


                     THE SIX CAROI PRINCIPLES
CAROI is based on six fundamental principles. CAROI strives to facilitate the resolution of
oversight findings; improve communication; foster collaboration; promote trust; develop
understanding and enhance performance. Adoption of these principles helps ensure optimal

                            Figure 1: Six Principles of CAROI

    improve audit                  improve                            foster 
      resolution                communication                     collaboration

                                    develop                         enhance 
    promote trust 
                                 understanding                    performance



The Office of Inspector General (OIG) of the United States Equal Employment Opportunity
Commission (EEOC) is conducting a pilot process called the Cooperative Audit Resolution and
Oversight Initiative (CAROI) [pronounced kuh-roy]. The OIG’s goal for this pilot program is to
use CAROI to improve resolution of recommendations contained in reports that require audit
follow-up. Since the OIG began using CAROI in 2012, we achieved significant results, including
improved communication with a key program office, and we identified an audit follow-up
process glitch. In this document we present an overview of the CAROI concept, why and how
the OIG implemented CAROI, and we discuss the benefits of our pilot program efforts.

CAROI Background

In 1995, the U.S. Department of Education (ED), Office of Inspector General, sought an
improved method for closing unresolved audit findings. Reasons for seeking a new method
included the failure to eliminate the causes of grant program deficiencies and the desire to avoid
lengthy legal battles over federal auditor’s findings that questioned how recipients spend the
grants. For example, one recipient of federal ED grants, the Commonwealth of Pennsylvania,
had a deficiency backlog of 119 matters. Some of the backlog dated back seven years. After
working with internal and external stakeholders, including state and local partners, ED developed
CAROI. Using CAROI, the ED OIG resolved the Pennsylvania backlog within six months.
CAROI is still in use by ED. Other examples of CAROI include ongoing efforts by the U.S.
Environmental Protection Agency (EPA) and the Office of Management and Budget (OMB). In
Massachusetts, CAROI is used not only to work with the federal government (e.g., ED and the
National Science Foundation) in resolving audits, but also within the state government by the
Department of Elementary and Secondary Education.

CAROI Fundamentals

CAROI seeks to improve dialogue, promote innovation in identifying solutions to problems,
foster continuous improvement of the audit process, improve the efficiency and effectiveness of
all oversight activities, and minimize bureaucratic and other inefficient methods. CAROI is well
suited to the OMB’s December 8, 2009 Open Government Directive M-10-06 (Directive). Both
the Directive and CAROI stress the importance of using innovative methods to increase
collaboration. The Directive requires federal agencies to take specific actions to implement the
principles of transparency, participation, and collaboration.

Challenges to successfully using CAROI for improving audit resolution include (1) individual
and organizational resistance to change and (2) identifying those opportunities for making audit
resolution a process built on cooperation, dialogue, and improvement of services.

The OIG’s Decision to Use CAROI

The OIG issues audit reports and other work products containing recommendations for the
EEOC. Auditees are required to address these recommendations using a written corrective action
plan (CAP) and other documentation, as appropriate, to demonstrate how they plan to address the
recommendations. The EEOC’s Office of Chief Financial Officer (OCFO) is the Audit Follow-
up Official (AFO), as stated in the relevant EEOC directive.

Over the past several years, we noted more than a few problems with audit resolution at the
EEOC, including but not limited to:

       Long-standing recommendations which have not been addressed in a timely manner;
       A lack of clarity in the audit resolution process; and
       A lack of meaningful communication between the AFO and the auditee regarding audit

In 2011, the OIG considered piloting CAROI to determine its viability as a tool to improve audit
follow-up at the EEOC. The OIG also decided to implement CAROI in a manner suited to the
EEOC’s organization and culture, and in line with the OIG’s resource constraints. One of
CAROI’s strengths is flexibility of implementation. With this flexibility and the OIG’s limited
staff and resources, the OIG decided to pilot CAROI in a measured manner, beginning with a
single product.

The OIG chose its Evaluation of the Management of the EEOC’s State and Local Programs
(Project Number 2010-09-AEP) completed in March 2011. We chose this work product for two
main reasons: (1) the managers of the audited entity, the Office of Field Programs (OFP), were
already familiar with the OIG conducting work in their program areas and (2) the number and
types of recommendations (moderately complex) would require ongoing and effective
communication to resolve.

Our Approach to Implementing CAROI

The OIG’s initial approach to implementing CAROI was more of an exploratory examination of
the conceptual underpinnings of the doctrine, rather than a formal test of its efficacy in real-
world applications. The implementation efforts were less scientific and structured in our
approach than outlined in the Association of Government Accountants’ CAROI Guide. Our
approach was not intended as a fundamental disregard for the CAROI Guide as much as it was an
attempt to determine how well we understood the six principles of CAROI, and whether the six
principles would work for a small agency in nontraditional audit conditions. Second, we had
concern that a more formal approach, coupled with a limited familiarity with CAROI could
create resistance to its’ implementation.

OIG felt that a low-keyed approach would reduce the risk of using an inordinate amount of
resources on an uncertain outcome. For example, we chose not to execute a CAROI agreement,
establish formal teams, or seek any specific outcomes (i.e., no specific goals were set for the
resolution of recommendations) other than to ascertain whether CAROI was applicable on a
broader scale. Our overarching goal was to gain fundamental knowledge about applying CAROI
and whether it could be an effective tool for the EEOC’s audit follow-up organization. Further,
we sought to help the auditee improve its experience with the audit resolution process, increase
their program efficiency and effectiveness, and improve their performance.

What We Learned

    1. CAROI is a valuable tool in identifying anomalies and systemic flaws in an
       organization’s audit follow-up and resolution process. CAROI can remedy such glitches,
       resulting in improved audit resolution program efficiency and effectiveness, and
       improvement in overall program operations.
    2. CAROI can reduce the potential for the auditee’s unwise investment of valuable
       resources in developing and implementing CAPs that are complex and perhaps
       misguided, or implementing CAPs designed to correct problems that do not exist or have
       been resolved.
    3. The CAROI paradigm is a preemptive and preventative audit resolution methodology. It
       is well suited to avert recurring instances of internal control failures, thereby improving
       the system of internal control and fostering greater trust and improved collaboration
       among and between the audit organization, the auditee, and the audit resolution function.
       The EEOC’s current approach to audit resolution relies heavily on the assumption that
       the auditee is knowledgeable about audit resolution.

Internal Stakeholder Involvement

In July 2012, the OIG met with the Office of Field Programs (OFP) management and managers
of the State and Local Programs to discuss the OIG’s CAROI initiative. Prior to the meeting, the
OIG provided program staff with information (e.g., information about CAPs) to prepare for a
substantive discussion of the principles and benefits of sound CAPs. During the meeting, the
principles of CAROI were presented and discussed, as well as the benefits of CAROI, and how
we envisioned developing and implementing CAROI.

As a result of the meeting, we determined that assisting the auditee with procedural and
formatting issues of its CAP would be the initial steps in implementing CAROI. Because of this
assistance and the auditee’s efforts, which included a good deal of dialogue, the auditee was able
to develop a well-structured CAP. It was then implemented, resulting in closure of all
recommendations in less time than would have been the case without the use of CAROI.

At this point, the OIG conducted interviews with the Director of the OFP and management and
staff of the State and Local Programs. The interviewees stated the following:

    1. Communication between the OFP and the OIG improved significantly. The OIG better
       understood the concerns of the OFP; and the OFP more fully understood the concerns of
       the OIG regarding improvements in program efficiency and effectiveness.
    2. The use of CAROI could result in improved program and activity results, due to a
       reduced focus on compliance and greater focus on results.
    3. The OIG should continue using CAROI because it may result in improved
       communication and enhanced program effectiveness and efficiency.
    4. Although implementing the CAROI pilot program took significant time, the investment
       of time was worthwhile, due to the benefits detailed above.

Additional Benefits from the CAROI Pilot

Because of the CAROI pilot program, the OIG identified an audit resolution process issue (the
delayed review of the CAP by the Audit Follow-up Official (AFO) involving the Evaluation of
the Management of the EEOC’s State and Local Programs). This discovery was the direct result
of the informal and flexible nature of CAROI (and a quality control check on the OIG’s
semiannual report revealing a difference between the OCFO’s and the OIG’s understanding of
the resolution of some recommendations).

The OIG then entered into discussions with the OFP and the OCFO to identify the cause of the
delayed review of the CAP. Together, these groups identified the cause and, more importantly,
confirmed the need to clarify the audit follow-up process and continue to communicate
throughout the process. Additional OCFO and the OIG staff meetings resulted in clarifications of
the audit follow-up process and agreement on the need for a memorandum of understanding to
institutionalize the audit follow-up process.

Prior to the end of our last reporting period (October 2013–March 2014), the OIG evaluation
staff and AFO staff discussed a CAP for another evaluation report, resulting in improved AFO
assessment of the CAP. In February and March of 2014, we met with the OCFO audit resolution
officials and discussed the CAP for the March 2013 OIG report EEOC Performance Measures.
The discussion, which centered on the structure of the CAP, led to clear and accelerated
communication between the AFO and the auditee. In turn, that resulted in clarification of the
auditee’s position on addressing several of the recommendations. This allowed the AFO to more
readily and objectively decide whether to close recommendations. We are pleased with the
additional transparency this has brought to the closeout process. In addition, the AFO reports that
CAROI has improved its interactions with the OIG, resulting in greater opportunities to discuss
the intent of recommendations, and whether proposed corrective actions adequately address
those recommendations.

We believe that consistent and more widespread use of CAROI will result in timelier disposition
of recommendations. This should lead to better products with more practical recommendations.

Next Steps

The OIG plans to use CAROI selectively on future evaluations and performance audits, and to
continue to provide results of the pilot program to, and solicit feedback from, EEOC
management and staff involved in audit follow-up. We also believe that CAROI may be useful
for the EEOC when resolving audit and related findings with oversight organizations other than
the OIG, such as the Government Accountability Office (GAO) and the U.S. Office of Personnel
Management. It should be noted that CAROI can be an effective resource for resolving audit and
related findings regardless of whether the audits are subject to the audit follow-up process.