OFFICE OF INSPECTOR GENER!L !udit Report Farm Credit !dministration’s Commissioning Program !-14-04 !uditor-in-Charge Tori Kaufman Issued March 31, 2015 F!RM CREDIT !DMINISTR!TION Farm Credit Administration Office of Inspector General 1501 Farm Credit Drive McLean, Virginia 22102-5090 March 31, 2015 The Honorable Kenneth A. Spearman, Board Chairman The Honorable Dallas P. Tonsager, Board Member The Honorable Jeffrey S. Hall, Board Member Farm Credit Administration 1501 Farm Credit Drive McLean, Virginia 22102‐5090 Dear Board Chairman Spearman and FCA Board Members Tonsager and Hall: The Office of Inspector General (OIG) completed an audit of FCA’s Commissioning Program. The objective of this audit was to determine whether FCA is effectively managing the Commissioning Program. During our audit, we found that the Office of Examination (OE) has a robust training and development program. Each pre‐commissioned examiner attends formal training courses, receives extensive on‐the‐job training, and completes certification testing. Progress is tracked and documented continuously throughout the multi‐year program. We identified nine opportunities for improvement. In response to our audit, OE and OMS have agreed to take actions to improve the Commissioning Program, including: 1. Identify and track specific commissioning costs to evaluate the cost of the program and identify cost‐saving opportunities and consider timekeeping code revisions, with OMS assistance in implementation. 2. Establish a process to verify time charged by Associate Examiners complies with work performed and timekeeping guidance. 3. Analyze the costs and benefits of streamlining and consolidating current testing and assessment milestones through the elimination of the final Commissioning Test simulations. 4. Establish a plan to compete Commissioning Program contractor services to manage risks of reliance on one source and ensure the best value to the Agency. 5. Ensure current Commissioning Program contracts are well‐defined in regards to general and administrative and hourly rates. 6. Ensure the invoice approval process for the Commissioning Program covers the requirements of the contract and review by all Agency personnel necessary to verify work performed before approval and payment. 7. Assess strategies to identify the cause of hiring shortfalls and employee attrition to meet commissioned examiner goals and maximize Agency investments. 8. Evaluate opportunities to implement Service Agreements or another type of comparable reimbursement arrangement to protect Agency investments in Commissioning Program training and certification. 9. Revise processes to provide feedback to every Associate Examiner on Technical Evaluations and Commissioning Test multiple‐choice test performance. We appreciate the courtesies and professionalism extended by FCA personnel to the OIG staff. If you have any questions about this audit, I would be pleased to meet with you at your convenience. Respectfully, Elizabeth M. Dean Inspector General Enclosure OBJECTIVE: To determine whether FCA is effectively managing the Commissioning Program. The Farm Credit Administration (FCA or Agency) has a robust training and development program to commission examiners. Each pre‐commissioned BACKGROUND: examiner (Associate Examiner) attends formal training courses, receives extensive A commission is a on‐the‐job training, and completes certification testing. Progress is tracked and designation to signify that documented continuously throughout the multi‐year program. Our review a Farm Credit revealed opportunities for improvement. Specifically: Administration examiner Overall costs of the Commissioning Program are not monitored, despite the is qualified to examine significance of the program and the level of resources committed. Farm Credit System Although there is a budget detailing employee staff days and travel days for institutions. The the Commissioning Program and contract services, the Agency’s Commissioning Program timekeeping and budgeting systems do not differentiate time or dollars consists of specific charged for commissioning from recruiting/hiring to commission training, on‐the‐job work certification. experience, and testing in order to cover necessary Potential efficiencies could be gained by streamlining and consolidating Office of Examination Commissioning Program assessment milestones. Associate Examiners must (OE) knowledge, skills, demonstrate specific competencies throughout a variety of continuous tasks, and performance comprehensive assessments to earn a commissioned examiner certification. attributes. Program requirements are completed over a multi‐ Contracts were single‐sourced to one contractor thereby, relying on one source to provide services that are critical to operating the Commissioning year, two‐tier process, Program puts the Agency at risk. which culminates with a commission and is the Staffing shortfalls and attrition impacted OE’s ability to maximize training foundation for OE’s investments in the Commissioning Program. Career Path Program. Commissioning Program Providing feedback to candidates for all components of Technical requirements, Evaluations and the Commissioning Test will support Associate Examiners’ responsibilities, policies, development. and procedures are described in Policies and There are 9 agreed‐upon actions to improve the Commissioning Program. Procedures Manual 503. Table of Contents BACKGROUND _______________________________________________________________________ 1 AUDIT RESULTS ______________________________________________________________________ 3 Opportunities for Improved Cost Monitoring ____________________________________________ 4 Agreed‐Upon Actions 1‐2 ____________________________________________________________ 5 Streamlining Testing and Evaluation ___________________________________________________ 5 Agreed‐Upon Action 3 _______________________________________________________________ 8 Contractor Oversight ________________________________________________________________ 8 Agreed‐Upon Actions 4‐6 ___________________________________________________________ 10 Other Areas for Improvement _______________________________________________________ 10 Agreed‐Upon Actions 7‐9 ___________________________________________________________ 11 OBJECTIVE, SCOPE, AND METHODOLOGY ________________________________________________ 12 ACRONYMS ________________________________________________________________________ 13 BACKGROUND The core mission of the Farm Credit Administration (FCA or Agency) is to regulate and supervise the Farm Credit System (FCS) in order to ensure a safe, sound and dependable source of credit and related services for agriculture and rural America. FCA’s Office of Examination (OE) plays a critical role in accomplishing this mission by examining and supervising FCS institutions. Examiners are the essential link with institutions through on‐site examinations, interactions with FCS boards and management, and written reports and correspondence. For this reason, one of OE’s top priorities is to maintain and train highly skilled examiners who understand the unique risks of agriculture, retain sufficient regulatory and financial experience, and communicate effectively. A commission is a designation to signify an FCA examiner has fulfilled professional, educational, and experience requirements and is qualified to examine FCS institutions. Specifically, only a commissioned examiner can act as an Examiner‐in‐Charge. The commission designation is used by other financial regulators, which makes it a valuable industry standard for the Agency. OE’s Commissioning Program consists of extensive formal training courses, targeted on‐the‐job work experience, and testing covering required knowledge, skills, tasks, and performance attributes (referred to as competencies). Pre‐commissioned examiners (Associate Examiners) achieve core competencies through training, and accomplishment is confirmed through testing. These required competencies are the entry way to promotion into OE’s Career Path Program and a career as an FCA examiner. The Commissioning Program is a multi‐year, two‐tier progression, which culminates with a commission. Commissioning Program Tier 1‐ Basic Tier 2‐Foundational •Years 1‐2 •Years 3 and beyond •Proficiency validated by •Commissioning Test certification Technical Evaluations completed Tier 1 includes years 1 and 2 of the Commissioning Program, and tier 2 includes years 3 and beyond. During tier 1, all Associate Examiners are assigned to OE’s Staff Development Division (SDD). Within SDD, Associate Examiners are assigned to trainers, experienced senior examiners who ensure work is consistent with quality standards and who provide extensive feedback. During tier 2, Associate Examiners are assigned to either the Market Risk Division or the Association Examination Division within OE. As part of their division assignments, they shadow a seasoned Examiner‐in‐Charge who is assigned as a coach. In addition to on‐the‐job training, many training courses must be completed at each level. This basic structure may be adjusted based on each Associate Examiner’s progress. 1 The Commissioning Program incorporates various tools to determine whether an Associate Examiner can demonstrate required competencies. Trainers and Examiners‐in‐Charge use the Associate Examiner Competency Evaluation System (AECES), a computer application developed in coordination with FCA’s Office of Management Services (OMS), to capture feedback and document successful completion of critical on‐the‐job tasks for each candidate. In addition, Associate Examiners’ progress is continuously evaluated through simulated training exercises that recreate work experiences, post‐training quizzes, and formal testing. Associate Examiners must pass three tests (described below) to advance through the Commissioning Program and achieve promotions. Technical Evaluation tests are administered at the end of years 1 and 2, before candidates are converted to permanent positions. These tests cover competencies developed during each respective year. Technical Technical Commissioning Evaluation 1 Evaluation 2 Test •Multiple‐choice test •Multiple‐choice test •Multiple‐choice test •Loan Simulation •Loan Simulation •Examination Scoping •Financial Institution and Planning Rating System Simulation Simulation •Loan Simulation (conducted offsite) •Examination Reporting Simulation (conducted offsite) Both Technical Evaluations include a multiple‐choice test and simulation tests. Technical Evaluation 1 includes a loan simulation, and Technical Evaluation 2 includes a loan simulation and a Financial Institution Rating System (FIRS) simulation. The objective of the loan simulations is to test the Associate Examiner’s ability to independently analyze risks, document support, and formulate interview questions for loan officers. The FIRS simulation measures the candidate’s skill in independently identifying an institution’s strengths and weaknesses, developing targeted questions for institution management, and drawing conclusions on the safety and soundness of an institution. Year 1 and 2 Technical Evaluations may also assist in identifying Associate Examiners who are not progressing as anticipated. Prior to year 3 a candidate falling short of anticipated progress can be released from the program in order to minimize further investments in their training and development. This assessment is increasingly important given the Agency’s small size and the resulting higher cost per candidate to administer formal training courses. The Commissioning Test is the final element of the program and serves as a standardized metric to evaluate candidates’ knowledge and demonstration of required performance attributes. Final Commissioning Test simulations test performance attributes including: planning exam activities, drawing conclusions, written and oral communication skills, and teamwork. Various controls are utilized to create an independent testing environment for these simulations. Multiple experienced FCA examiners (assessors) score each candidate and role play as loan officers and institution officials. Assessors are not assigned to candidates they have supervised or worked closely with, and a contractor oversees each assessor to evaluate objectivity. In addition, the composite score necessary to achieve a passing rate is 2 not disseminated as a further measure to ensure valid scoring. Following the final multiple‐choice test and three days of final simulation testing, if passed, a candidate is certified as a commissioned FCA examiner. In 2014, OE hired a contractor to analyze and update the required competencies for newly commissioned examiners. The same contractor had completed a prior review and update as part of OE’s 2008 Commissioning Program redesign project. The redesign project also allocated responsibilities to OMS, including: Assisting in administration of the Commissioning Test twice a year, Grading the multiple‐choice and reviewing the simulation Commissioning Test results, and Funding a contractor to monitor each final offsite Commissioning Test simulation. Responsibilities and requirements of the Commissioning Program are described within OE’s Policies and Procedures Manual (PPM) 503, Examiner Commissioning. AUDIT RESULTS OE has a robust training and development program to ensure Associate Examiners obtain the necessary competencies to examine FCS institutions. Formal training courses are administered over about a 3‐year period and cover areas such as credit, exam management, finance, and operations. Commissioning Program requirements are aligned with essential OE knowledge, skills, tasks, and performance attributes, which are updated periodically to reflect current examination responsibilities. OE also reviews testing and training materials each year for comprehensiveness. Staff levels and the number of commissioned examiners are continuously tracked through OE quarterly reports, yearly operating plans, briefings to the FCA Board, and the Agency Human Capital Plan. Several controls are in place to ensure Associate Examiners meet developmental milestones. Performance of critical on‐the‐job activities is rated and tracked for each candidate throughout the program. The Commissioning Program’s tiered approach with year 1 and 2 testing is also designed to assess the level at which Associate Examiners can demonstrate required competencies. Promotion to higher grade levels is contingent on passing each test. This structure limits Agency investment based on specific progress. Our review revealed opportunities for improvement and efficiency in the Commissioning Program. Specifically, we identified opportunities to: improve cost monitoring, streamline testing and evaluation, improve contractor oversight, reduce staffing shortfalls, and provide additional feedback to candidates. 3 Opportunities for Improved Cost Monitoring Commissioning Cost Estimate The significance of the Commissioning Program, and the resources committed to its success, warrant increased cost monitoring. The Government Accountability Office (GAO) created A Guide for Assessing Strategic Training and Development Efforts in the Federal Government (GAO‐04‐546G) as a framework to address how agencies plan, design, implement, and evaluate effective training and development programs. The guide highlights the importance of knowing costs to improve planning, design/development, implementation, and evaluation of training programs. By capturing the costs of these programs, an agency can determine the appropriate level of investment and prioritize funding to address the most important needs first. Cost information also informs managerial decisions by evaluating the cost‐effectiveness of different options for achieving learning objectives. The cost of the FCA Commissioning Program is not tracked in a way that could be utilized to identify the cost of the program from recruiting/hiring to obtaining the commission certification and evaluate cost‐ saving opportunities. OE formulates budgets and tracks costs based on employee staff days and the cost of travel, its main cost drivers. Total OE staff days and travel costs are monitored in general, activity‐based categories that correspond to timekeeping codes, and the Agency’s timekeeping and budgeting systems do not sufficiently allow the identification and differentiation of various commissioning activities. Currently, the cost to hire, train, and commission an Associate Examiner is included in five different timekeeping categories: Recruiting, Training Administration and Development, Training Taken, On‐the‐Job Training, and Commissioning Program For the purposes of this audit, OE prepared a general estimate of the total cost of the Commissioning Program and the cost to commission an individual Associate Examiner. OE applied percentages of budgeted OE staff days to total dollar costs for each timekeeping category and used these totals to calculate the average total cost for the Commissioning Program from FY 2011 through 2014. This average total cost per year was divided by the average number of participating Associate Examiners per year and multiplied by four to reflect the general time period to be commissioned. OE estimated the cost to commission an Associate Examiner to be about $416,323. OE Estimated Commissioning Costs Average Commissioning Program Cost per year $4,163,235 Average Number of Associate Examiners 40 Average Total Cost Per Year/Average Number of Associate Examiners $104,081 Four Year Estimated Average Cost per Associate Examiner $416,3231 1 This OE estimate was based on a number of assumptions and manual adjustments. OIG did not validate its reliability. This estimate does not include OMS costs, contractor costs, or hotel space rental expenses associated with the Commissioning Program. In addition, OMS performance reports did not differentiate “On‐The‐Job Training” time, so actual dollar costs in this category were assumed to be the same as what was budgeted. 4 Timekeeping We also noted disconnects in timekeeping, timekeeping guidelines, and budgeting that further limited the accuracy of commissioning cost monitoring. For example: For the first two years of the Commissioning Program (tier 1) Associate Examiners were directed by timekeeping guidance to code all institution examination hours as “On‐The‐Job Training.” However, for FY 2013 and 2014, 6 tier 1 Associate Examiners charged about 190 onsite examination staff days incorrectly. OE officials stated that despite the direction in timekeeping guidance, a percentage of tier 1 Associate Examiners’ time directly contributed to actual institution examination versus on‐the‐job training. In regards to budgeting, OE’s Guide for Recording Time and Travel directed Associate Examiners to charge time spent studying for Technical Evaluations and the Commissioning Test to the “Commissioning Program” timekeeping code. However, this time was consistently budgeted as “Training Taken.” OE officials stated that this issue stemmed from an ongoing disconnect between timekeeping guidance and the way OE budgeted for study time. Agreed‐Upon Actions 1‐2 To improve cost monitoring, OE agreed to: 1. Identify and track specific commissioning costs to evaluate the cost of the program and identify cost‐saving opportunities and consider timekeeping code revisions, with OMS assistance in implementation. 2. Establish a process to verify time charged by Associate Examiners complies with work performed and timekeeping guidance. Streamlining Testing and Evaluation Opportunities for Streamlining and Consolidating OE utilizes the Commissioning Test as a standardized metric to certify Associate Examiners can demonstrate competencies of a commissioned examiner. Before the Commissioning Test, each candidate must successfully complete various assessment milestones, including: simulated exercises as part of training, post‐training quizzes, year 1 and 2 Technical Evaluation tests, and fully‐effective ratings for all AECES on‐the‐job tasks. OE’s training program is designed to assist Associate Examiners in reaching these milestones. OE developed its competency‐based Commissioning Program to cover knowledge, skills, tasks, and performance attributes necessary to be a commissioned examiner. These competencies include numerous aspects of credit, exam management, finance, and operations. In total, the Agency identified more than 400 interrelated competencies required to be a commissioned examiner. 5 Throughout the Commissioning Program, each candidate is continuously evaluated to ensure they have the capacity to demonstrate these competencies. Of the more than 400 competencies, OE identified nearly 150 critical on‐the‐job tasks. Each time an Associate Examiner attempts one of these on‐the‐job tasks it is tracked in AECES, and it is rated as ineffective, moderately effective, or effective based on specific definitions related to adequate performance. Trainers and Examiners‐in‐Charge also provide feedback and comments, and must do so if a task is rated ineffectively. In order to sit for the Commissioning Test, an Associate Examiner must have effectively completed all on‐the‐job tasks tracked in AECES. We noted potential efficiencies could be gained by streamlining the Commissioning Program, perhaps by condensing and consolidating assessment milestones. A large portion of the Commissioning Test consists of three simulations (Examination Scoping and Planning, Loans, and Examination Reporting) graded by independent assessors, selected, trained and observed to create a controlled environment. Because other existing assessment channels may provide assurance that an Associate Examiner is proficient in mastering these competencies, OE could potentially revise the program to decrease costs and reallocate resources. OE considers simulations to be a necessary standardized metric to assess and certify performance attributes. While the final Commissioning Test simulations create a standardized metric, standardization is also accomplished through the competency‐based training model utilized by OE. For example, completion of the same 150 tasks is tracked for each Associate Examiner in AECES, and effective ratings are based on specific definitions. The chart below depicts examples of potential areas for consolidation: Performance Attributes Evaluated in Examples of Critical Tasks Evaluated in Final Commissioning Test Simulations AECES Delegate examination responsibilities to Planning oversight and examination examiners and develop examination activities scope, strategies, and schedules Evaluate underlying causes of capital Analyzing information, drawing trends and evaluate the amount, conclusions, researching, and investigating composition, and stability of earnings Interview loan officers to obtain or verify Oral communication loan information Produce written products that adhere to Written communication the general writing protocol Serve as Examiner‐in‐Charge for institution Teamwork examinations or lead examiner for examination segments 6 Final Commissioning Test Simulation Resource Requirements Planning and conducting final Commissioning Test simulations is very resource‐intensive. Final Commissioning Test simulations, held twice a year, are a weeklong process each time. Two of the three final simulations are conducted at an offsite hotel, attended and monitored by a contractor. In addition to the Associate Examiners undergoing the simulation, for the years in review, each final offsite Commissioning Test simulation was attended by 2‐4 monitors, and 6‐12 experienced FCA examiners assigned to role play and grade each test (assessors). All assessors are also trained by a contractor, both when selected to be an assessor, and before the actual simulations. For FY 2011 through 2014, about $289,303 was spent to conduct final offsite Commissioning Test simulations, which included travel, contractor, and hotel expenses, as noted in the chart below. Final Offsite Commissioning Test Simulation Costs Travel Contractor Hotel Total Candidates Test Date Monitors Assessors Costs Cost Costs Cost Tested October $18,295 $7,725 $3,229 $29,248 4 6 5 2010 October $28,328 $8,529 $14,742 $51,599 3 9 8 2011* April 2012 $19,886 $11,429 $9,194 $40,509 3 6 5 October $39,254 $11,179 $9,019 $59,452 3 12 10 2012 April 2013 $20,903 $10,261 $7,922 $39,087 3 6 6 October $29,174 $10,874 $1,536 $41,584 3 9 7 2013 April 2014 $17,595 $7,622 $2,608 $27,825 2 6 4 Total $173,434 $67,620 $48,250 $289,303 Average Cost Per $24,776 $9,660 $6,893 $41,329 Test *Commissioning Test simulations were not held in April 2011. In addition to reducing costs by eliminating final offsite Commissioning Test simulations, staff time could be re‐directed to other mission‐based activities. In order to quantify this, we calculated the percentages of budgeted staff days to administer final Commissioning Test simulations: Staff Time for Final Commissioning Test Simulations FY 2015 Budget FY 2016 Budget Staff Days for Final Commissioning Test Simulations 216 130 Overall “Commissioning Program” Staff Days 772 590 Simulation Percentage of “Commissioning Program” Staff Days 28% 22% Average Simulation Percentage 25% 2 2 The actual average simulation percentage is 25.0066% and was rounded for simplification. 7 We then applied this percentage to actual “Commissioning Program” dollar costs, which tracked overall testing costs for OE. Costs for Staff Time FY 2011 FY 2012 FY 2013 FY 2014 Total “Commissioning Program” Dollar $231,392 $311,207 $369,379 $345,592 $1,257,570 Costs Average Simulation Percentage $57,863 $77,822 $92,369 $86,421 $314,475 Average Simulation Cost Per Year $78,619 Considering $41,329 as the average cost of final offsite simulations and the simulation is given twice a year, we calculated that about $82,658 per year in expenses could be eliminated by removing final Commissioning Test simulations. In addition, the calculated value of time spent administering these final simulations that could be re‐directed under a revised testing structure would average about $78,619 per year. In total, we conservatively estimated3 that the Agency could recognize about $161,277 in savings each year by removing final Commissioning Test simulations. Although there would be resource requirements associated with revising existing assessments and substituting the same quality of oversight and testing to be given at earlier time junctures, a review process is already in place during 2015 to update the Commissioning Program based on the September 2014 revised competencies. In addition to decreased costs and resource requirements, a condensed testing structure could also decrease the timeframe to commission an examiner. Competencies are developed and demonstrated throughout the Commissioning Program. The majority of Associate Examiners pass the multiple‐choice portion of the Commissioning Test after their first attempt. Additionally, the multiple‐choice portion of the Commissioning Test is administered in the Associate Examiner’s field office location, so it would be easier and less costly to re‐schedule, if necessary. Although these costs and benefits represent examples, OE expertise will provide additional substantial opportunities to improve program effectiveness. Agreed‐Upon Action 3 To improve the Commissioning Program, OE agreed to: 3. Analyze the costs and benefits of streamlining and consolidating current testing and assessment milestones through the elimination of the final Commissioning Test simulations. Contractor Oversight OMS and OE both used the same contractor for consulting services related to the Commissioning Program. This contractor has provided consulting services on the Agency’s training and commissioning certification programs for more than 20 years. We found that contractor oversight could be improved. Specifically: 3 We believe our estimate is conservative because in calculating the staff days for final Commissioning Test simulations, we only included the days budgeted for administering the test. We did not include time for reviewing final Commissioning Test simulations, planning new exam exercises, or initial training for assessors. 8 All work was awarded to the contractor on a single‐source basis. A single‐source selection is the award of a contract without competition and requires a documented justification. FCA’s single source justification required an informal market survey or a description of why it was not conducted. OE and OMS stated a market survey was not conducted due to the contractor’s “successful history as a consultant on the Agency’s Commissioning Program.” The OMS contract involved OE work on the Commissioning Program; however, many invoices were only reviewed by OMS. Given the contractor’s involvement with both OE and OMS for the Commissioning Program, it is important to ensure invoices are routed to the appropriate individuals for review. We identified a few instances where the contractor was reimbursed outside of specific contract allowable charges. For instance, the contractor was reimbursed for general and administrative expenses for one trip that were not allowed because a rate had not been specified in the contract. Also, the contractor increased hourly rates for two individuals without prior approval. The contract identified estimated overall costs based on amounts quoted in the contractor’s proposal but did not identify hourly rates for each staff member. The contractor stated that the revised hourly rates were based on merit and market adjustment increases. However, the contractor should have contacted FCA’s Contracting Officer and requested the increase or charged the same rates throughout the period of performance. Lastly, in three instances, the contractor was reimbursed for meals and incidentals included on lodging receipts in addition to receiving per diem allotments. The contractor was also reimbursed for lodging expenses that exceeded the allowable per diem maximum for one trip. OE and OMS justified the single‐source selection based on “logical follow‐on” and the contractor’s longstanding history and experience with the Commissioning Program. This contractor was awarded 6 contracts for a wide variety of services: test monitoring, analyzing required competencies, and developing hiring assessments. This brings into question whether each award is a “logical follow‐on” for a continuation of previous services. In addition, relying on a single contractor, and actually one individual for certain services, puts the Agency at risk if this contractor is not available. It is important to ensure there is more than one source capable of providing services that are critical to ongoing operation of the Commissioning Program. Also, competition ensures the best value for the Agency and avoids the appearance of preference. Each contract had the same Contracting Officer’s Technical Representative (COTR). The COTR sent invoices to OE and OMS personnel to ensure that the work was performed. Upon receiving an email that they were satisfied with the invoice, the COTR approved invoices for payment. OE and OMS officials who reviewed the contractor’s invoices stated that their review was “high‐level” and they did not receive training specific to invoice review or approval. PPM 840, Role and Responsibility of the Contracting Officer’s Technical Representative, states that the COTR may solicit assistance of other agency personnel during the performance period of the contract, 9 but the COTR is responsible for monitoring contractor performance to ensure it is in line with the contract. Agreed‐Upon Actions 4‐6 To improve contract oversight and administration, OE and OMS agreed to: 4. Establish a plan to compete Commissioning Program contractor services to manage risks of reliance on one source and ensure the best value to the Agency. 5. Ensure current Commissioning Program contracts are well‐defined in regards to general and administrative and hourly rates. 6. Ensure the invoice approval process for the Commissioning Program covers the requirements of the contract and review by all Agency personnel necessary to verify work performed before approval and payment. Other Areas for Improvement Maximizing Training Investments FCA’s pending wave of retirements and OE’s position as the primary human capital conduit for the Agency make the success of the Commissioning Program increasingly important. In FCA’s 2013‐2017 Human Capital Plan, OE projected staff levels would build to 180 over the five‐year period. OE determined a need for about 125‐130 of this total to be commissioned examiners and technical specialists. OE originally aimed to hire 10 Associate Examiners each year, but this goal was increased to account for hiring shortfalls and increased attrition. Planned vs. Actual Hiring, Attrition and Commissioning FY 2011 FY 2012 FY 2013 FY 2014 Planned Associate Examiner Hiring 13 10 15 15 Actual Associate Examiner Hiring 13 6 11 9 Planned Associate Examiner Attrition Rate 11% 8% 5% 10% Actual Associate Examiner Attrition Rate 15% 13% 5% 13% Planned Newly Commissioned Examiners 5 12 12 9 Actual Newly Commissioned Examiners 3 9 8 8 As a smaller financial regulator, it is important for FCA to maximize training investments. Because training course costs are mostly fixed each year, hiring shortfalls result in a higher per capita training cost. In other words, 9 Associate Examiners or 15 require relatively the same cost for classroom instruction. 10 OE largely attributed hiring shortfalls to challenges in using the OPM‐required Pathways Program, a hiring program for students and recent graduates implemented in 2012. OE addressed this challenge by revising its hiring assessment questionnaire and online tests for Associate Examiners to further limit the pool of applicants to qualified candidates. Nine of the planned 15 Associate Examiners were hired during FY 2014 after implementing the new method. As a result, more hiring will be necessary in the future to meet planned projections. If an Associate Examiner leaves the Agency during the Commissioning Program, it also affects training investments. Currently, OE does not have a reimbursement opportunity or other method to protect FCA’s investments if an Associate Examiner or newly commissioned examiner leaves the Agency. By contrast, FCA’s training policy, PPM 843 Training, Development, Professional Certifications, Licenses, and Membership Fees, requires employees to sign a 1‐year Service Agreement when receiving more than 80 hours of training for one course or $15,000 tuition for a particular subject matter in a 12‐month period. If an employee voluntarily leaves the Agency before completing this service, the employee will reimburse FCA for tuition (unless the obligation is waived). Although the Commissioning Program does not specifically fit this definition, this type of arrangement with required reimbursement could be extended to the Commissioning Program to limit the risk of costly attrition after training and certification. Feedback Feedback is another important aspect of an Associate Examiner’s testing and development. OE’s Performance Feedback Process directive emphasizes the importance of continuous, constructive feedback to address individual performance and identify progress towards career goals. Through interviews, we learned that OE did not administer feedback to all candidates for components of Technical Evaluations and the Commissioning Test. For the Technical Evaluations, there is a pass/fail score. Associate Examiners receive feedback only if they do not pass any portion of the test. For the Commissioning Test, candidates receive simulation feedback regardless of whether or not they pass, but feedback is not provided for the multiple‐choice tests. Because the Technical Evaluations are designed to represent condensed versions of the Commissioning Test and serve as a preparation tool, feedback would be useful to focus on strengths and weaknesses regardless of whether or not the candidate passes. Candidates informed the OIG that additional feedback would be put to good use in preparing for the next phase of their certification. Agreed‐Upon Actions 7‐9 To support human capital requirements and Associate Examiner development, OE and OMS agreed to: 7. Assess strategies to identify the cause of hiring shortfalls and employee attrition to meet commissioned examiner goals and maximize Agency investments. 8. Evaluate opportunities to implement Service Agreements or another type of comparable reimbursement arrangement to protect Agency investments in Commissioning Program training and certification. 9. Revise processes to provide feedback to every Associate Examiner on Technical Evaluations and Commissioning Test multiple‐choice test performance. 11 OBJECTIVE, SCOPE, AND METHODOLOGY The objective of this audit was to determine whether FCA is effectively managing the Commissioning Program. We conducted fieldwork at FCA’s headquarters in McLean, VA from September 2014 through March 2015. We limited our scope to Fiscal Years 2011 – 2014. We completed the following steps to accomplish the objective: Reviewed applicable laws, regulations, policies and procedures related to the Commissioning Program. Interviewed OE and OMS officials on Commissioning Program plans, programs, and processes. Obtained and analyzed Human Capital Plans, Operating Plans, Quarterly Reports, and presentations to FCA’s Board. Evaluated FY 2015‐2016 Commissioning Program budget documentation and how costs and resources are monitored. Reviewed estimated Commissioning Program costs prepared by OE and the cost to commission an Associate Examiner over four years. Analyzed the costs of final Commissioning Test simulations. Analyzed and reviewed contractor invoices pertaining to Commissioning Program administration services and reviewed single‐source justifications for the associated contracts. Interviewed a judgmental sample of six Associate Examiners in different stages of the Commissioning Program. This audit was performed in accordance with Generally Accepted Government Auditing Standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We assessed internal controls and compliance with laws and regulations to the extent necessary to satisfy the objective. Our review would not necessarily have disclosed all internal control deficiencies that may have existed at the time of our audit. We assessed the computer‐processed data relevant to our audit objective and identified issues included in the body of the report and footnotes. Overall, we believe the evidence obtained provides a reasonable basis for our conclusions based on our audit objective. 12 ACRONYMS AECES Associate Examiner Competency Evaluation System COTR Contracting Officer’s Technical Representative FCA Farm Credit Administration FCS Farm Credit System GAO U.S. Government Accountability Office OE Office of Examination OIG Office of Inspector General OMS Office of Management Services PPM Policies and Procedures Manual SDD Staff Development Division 13 R E P O R T Fraud | Waste | Abuse | Mismanagement FARM CREDIT ADMINISTRATION OFFICE OF INSPECTOR GENERAL Phone: Toll Free (800) 437‐7322; (703) 883‐4316 Fax: (703) 883‐4059 E‐mail: fca‐ig‐firstname.lastname@example.org Mail: Farm Credit Administration Office of Inspector General 1501 Farm Credit Drive McLean, VA 22102‐5090
FCA's Commissioning Program
Published by the Farm Credit Administration, Office of Inspector General on 2015-03-31.
Below is a raw (and likely hideous) rendition of the original report. (PDF)