U.S. Department of Education Office of Inspector General American Recovery and Reinvestment Act of 2009 The Department’s Process for Screening and Selecting Peer Reviewers for the Race to the Top Grant Program Final Audit Report ED-OIG/A19K0006 August 2010 UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF INSPECTOR GENERAL August 16, 2010 Thelma Meléndez de Santa Ana, Ph.D. Assistant Secretary Office of Elementary and Secondary Education U.S. Department of Education 400 Maryland Avenue, S.W. Washington, DC 20202 Dear Dr. Meléndez de Santa Ana: This final audit report presents the results of our audit of the U.S. Department of Education’s process for screening and selecting peer reviewers for Phase 1 of the Race to the Top discretionary grant program competition. We received the Office of Elementary and Secondary Education’s comments on the contents of our draft report. The comments are summarized within the Results section of this report. Corrective actions proposed (resolution phase) and implemented (closure phase) by your office will be monitored and tracked through the Department’s Audit Accountability and Resolution Tracking System (AARTS). Department policy requires that you develop a final corrective action plan (CAP) for our review in the automated system within 30 days of the issuance of this report. The CAP should set forth the specific action items, and targeted completion dates, necessary to implement final corrective actions on the findings and recommendations contained in this final audit report. In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector General is required to report to Congress twice a year on the audits that remain unresolved after 6 months from the date of issuance. In accordance with the Freedom of Information Act (5 U.S.C. § 552), reports issued by the Office of Inspector General are available to members of the press and general public to the extent information contained therein is not subject to exemptions in the Act. We appreciate the cooperation given to us during this review. If you have any questions, please call Michele Weaver-Dugan at (202) 245-6941. Sincerely, Keith West /s/ Assistant Inspector General for Audit The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational excellence and ensuring equal access. Final Audit Report Page 3 of 8 ED-OIG/A19K0006 The Department’s Process for Screening and Selecting Peer Reviewers for the Race to the Top Grant Program Control Number ED-OIG/A19K0006 PURPOSE The American Recovery and Reinvestment Act of 2009 (ARRA) places a heavy emphasis on accountability and transparency, and in doing so, increases the responsibilities of the agencies that are impacted by the Act. Overall, the U.S. Department of Education (Department) is responsible for ensuring that education-related ARRA funds reach intended recipients and achieve intended results. The Department’s responsibilities further include ensuring that competitive discretionary grant review processes are consistent with applicable laws and regulations and are executed in a manner that is both fair and objective. As a general rule, the Department’s policy is that a fair and competitive review process is enhanced by the use of outside reviewers to provide an independent perspective. This final report provides the results of our audit of the Department’s process for screening and selecting peer reviewers for Phase 1 of the Race to the Top (RTT) discretionary grant program competition. This $4 billion program represents the largest-ever single Federal investment in school reform and improvement efforts. The objectives of our audit were to determine the: 1. Appropriateness of Department actions in the screening and selection of peer reviewers for the RTT discretionary grant program competition (including compliance with applicable laws, regulations, policies, and procedures); and 2. Effectiveness of the Department’s processes and controls in identifying conflicts of interest or other issues. RESULTS We found that the Department’s process for screening and selecting peer reviewers for Phase 1 of the RTT discretionary grant program competition was generally appropriate and effective in identifying applicant conflicts of interest.1 The Department employed a 10-step selection process that involved no less than 35 Department employees and included at least 4 separate checks for conflicts of interest, performed by personnel from 3 distinct entities – the Office of Elementary and Secondary Education (OESE), the Office of the General Counsel (OGC), and a private contractor.2 Applicants were screened for both direct and indirect conflicts of interest,3 and in instances where such conflicts were identified, either dropped from consideration and informed of the reason(s) or granted a waiver for indirect conflicts to participate in the RTT Phase 1 application review process. The Department granted waivers only for indirect conflicts 1 A narrative overview of the Department’s implemented process is included as Attachment 1 to this report. A summary of participants and outcomes of each step of the implemented process is included as Attachment 2. 2 The group of 35 individuals was comprised of 28 career employees and 7 political appointees from 9 principal offices. See Attachment 2 for additional detail on the employees and offices involved in the RTT peer reviewer screening and selection process. 3 A direct conflict of interest is a situation in which an individual’s financial interests would be affected by the outcome of the competition. An indirect conflict of interest includes relationships with entities or individuals who might stand to benefit financially from the outcome of the competition. Final Audit Report Page 4 of 8 ED-OIG/A19K0006 determined not to be substantial. Overall, 15 of the 49 individuals (31 percent) who reviewed RTT Phase 1 applications received waivers from the Department. We reviewed the waivers granted as part of our audit and found them to be granted and issued in accordance with the Department’s policies and procedures.4 However, we determined that the Department did not perform a check of selected RTT peer reviewers against the General Services Administration’s (GSA) Excluded Parties List System (EPLS) or adequately document formal approval of its peer reviewer roster prior to the beginning of the application review process.5 This verification and documentation occurred after the initial application review and rating was completed and the Department publically announced the 16 RTT finalists. Although no RTT peer reviewers appeared in the EPLS, the integrity of the review process could have been compromised had it been discovered that a reviewer – particularly one involved in such a high visibility program – was debarred or suspended from doing business with the Federal government. In its response to the draft audit report, OESE concurred that there was an issue with the timeliness of the EPLS verification. However, OESE believed the issue did not impact the integrity or quality of either the competition or the review process and further believed the issue should not be reported as an audit finding. While we acknowledge that no RTT peer reviewers were found in the EPLS, we do not agree with OESE’s position on the issue. The 16 RTT finalists had been publicly announced prior to the EPLS verification. We believe that due to this timing, the integrity of the review process could have been compromised had any reviewers been identified in the EPLS. OESE’s comments are summarized at the end of the finding. The full text of OESE’s response is included as Attachment 4 to this report. BACKGROUND The RTT program is a discretionary grant program, new in fiscal year 2010, authorized under the ARRA. It consists of two separate grant programs: (1) RTT assessment grants, for which $350 million has been set aside for the purpose of supporting States in the development of a next generation of assessments aligned to common sets of standards, and (2) RTT competitive State grants for reform, valued cumulatively at $4 billion and the focus of this audit. The Department decided to make RTT grant awards in two phases. Phase 1 applications were due on January 19, 2010 and awards were announced on March 29, 2010. Phase 2 applications were due on June 1, 2010, with awards expected by September 30, 2010. The purpose of the RTT program is to encourage and reward States that are creating the conditions for education innovation and reform; achieving significant improvement in student outcomes, including making substantial gains in student achievement, closing achievement gaps, 4 A document posted online by the Department entitled Race to the Top Summary of Conflict of Interest Policy and Procedure states that “A waiver may be issued when a determination is made that an individual’s financial interest is not so substantial as to affect the integrity of his or her services and, given all relevant facts, the Department’s need for the individual reviewer’s services outweighs the possibility that a reasonable person would question the reviewer’s impartiality.” 5 The purpose of the EPLS is to provide a single comprehensive list of individuals and firms excluded by Federal government agencies from receiving Federal contracts or Federally-approved subcontracts and from certain types of Federal financial and nonfinancial assistance and benefits. Final Audit Report Page 5 of 8 ED-OIG/A19K0006 improving high school graduation rates, and ensuring student preparation for success in college and careers; and implementing ambitious plans in four core education reform areas: Adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global economy; Building data systems that measure student growth and success, and inform teachers and principals about how they can improve instruction; Recruiting, developing, rewarding, and retaining effective teachers and principals, especially where they are needed most; and Turning around our lowest-achieving schools. In November 2009, the Department published a notice of final priorities in the Federal Register and invited applications for RTT grants.6 The notice provided a brief description of the application review process, stating that a two-tiered review process would be used to judge the eligible applications. The initial review would be based solely on reviews of the written applications, while the final review would be based on both a State’s written application and in- person presentation. The notice also stated that the Department would use independent reviewers chosen from a pool of qualified educators, scholars, and other individuals knowledgeable in education reform. It further noted that the Department would thoroughly screen all reviewers for conflicts of interest to ensure a fair and competitive review process. To facilitate the selection of RTT peer reviewers, the Department posted a Dear Colleague Letter from the Secretary on August 31, 2009. The letter stated that the Department sought 50 to 80 individuals to serve on the peer review panels, each with expertise in some, if not all, of the following areas: (1) education policy, (2) education reform, (3) capacity and scale, and (4) application review and evaluation. It added that any applicant's selection as a peer reviewer for the RTT competition would also include a review for possible, apparent, and/or actual conflicts of interest, and that if a potential conflict of interest was identified, the Department would consider whether the applicant could participate as a peer reviewer in full compliance with all applicable Department policies and procedures. As part of this process, each applicant would be required to complete a conflict of interest questionnaire that provided information about the applicant’s professional experience(s), including any financial interest that the applicant may have in any State's RTT application. The Department planned to provide each peer reviewer with an honorarium of up to $7,500 for their work. On January 25, 2010, the Department posted a description of the RTT application review process on its website. This included information on the screening and selection of peer reviewers. The description stated that the Department’s legal ethics team eliminated any applicant with existing or potential conflicts of interest, including people currently employed by a State department of education or school district. It further stated that, despite the extensive vetting that occurred prior to the selection of the 58 reviewers, the Department recognized that in the process of reading an application, a reviewer may spot a potential conflict that had not been considered. If such conflicts occurred, applications would be reassigned among reviewers. To mitigate this, reviewers would also be prohibited from reviewing applications from their home State or States where they had any potential conflicts of interest. 6 A draft notice was put forth for public review and comment in July 2009. Final Audit Report Page 6 of 8 ED-OIG/A19K0006 FINDING The Department Did Not Perform Timely EPLS Verification or Adequately Document Approval of its Peer Reviewer Roster We determined that OESE did not perform a check of selected RTT peer reviewers against the GSA’s EPLS prior to the beginning of the application review process and selection of the 16 RTT finalists. Individual RTT program application reviews began on January 23, 2010, and RTT finalists were announced on March 4, 2010. OESE staff indicated that the debarment check was conducted by their front office during the week of March 8, 2010, as evidenced by a related memorandum signed by their Executive Officer on March 9, 2010. In addition, we found OESE did not maintain required documentation in the grant files to support that the peer reviewer roster was officially approved prior to the start of the application review process. OESE noted that the Executive Officer’s March 9, 2010 correspondence is the standard documentation used as evidence that the overall list of peer reviewers is approved. OESE emphasized that these reviewers were actually approved during a meeting of RTT and OESE leadership that took place on January 5, 2010. OESE cited two documents found during our review of the Grant Program Competition File as evidence that the reviewers were approved on January 5: (1) a meeting invitation, dated January 5, 2010, that identified members of the RTT Program, Leadership, and Management Teams as required or optional attendees;7 and (2) an adjacent list of 74 peer reviewers. However, neither document was signed or otherwise indicated approval of the reviewer roster prior to the start of the application review process on January 23, 2010. Section 3.4.3 of the Department’s Handbook for the Discretionary Grant Process states The program staff must also ensure that the names of any reviewers to be chosen do not appear on the GSA Excluded Parties List System (EPLS) list of persons who have been debarred or suspended. … After reviewers are selected, the program staff submits the reviewer roster for each competition to the program official for review and approval. The program official must approve the reviewer roster and include it as part of the Grant Program Competition File prior to beginning the review. 7 The Department established four working groups to guide and direct the RTT program, each consisting of representatives from various principal offices. The Response Team convenes frequently to respond to questions on the RTT application. The Program Team convenes as-needed to address programmatic questions and make recommendations to the Leadership Team. The Leadership Team convenes weekly or biweekly to consider recommendations made by the Program Team and discuss other programmatic questions, as needed. Lastly, the Management Team, which consists of the RTT Director, the Director of OESE’s Academic Improvement and Teacher Quality programs, and the Senior Advisor for Policy and Programs in the Office of the Deputy Secretary, meets frequently, as needed, to consider and make decisions pertaining to RTT programmatic and policy issues. The Application Technical Review Plan (ATRP) states that these three Management Team officials have had central roles in the planning, development, and implementation processes for the RTT program and competition and have participated as members of the Leadership Team. Final Audit Report Page 7 of 8 ED-OIG/A19K0006 We found that OESE staff did not provide a memorandum that served as both a debarment check request and a request for approval of prospective peer reviewers to their Deputy Assistant Secretary for Management until March 2, 2010. As a result, the EPLS check was not completed and official roster approval was not documented until 45 days after the start of the application review process and 5 days after the announcement of the finalists. As outlined in Attachment 2, the screening and selection process for RTT peer reviewers otherwise appeared detailed and thorough. We confirmed that none of the 588 selected peer reviewers were, in fact, in the EPLS. However, we identified four reviewers for whom additional research was required, including one who shared the same name and location as an individual in the EPLS. Failure to perform a timely EPLS verification and document roster approval could have compromised the integrity of the review process, especially had it been discovered that a reviewer – particularly one involved in such a high visibility program – was debarred or suspended from doing business with the Federal government after the 16 RTT finalists were announced. Recommendation We recommend that the Assistant Secretary for OESE: 1.1 Ensure that timely EPLS verifications are performed and that reviewer rosters are approved and placed in the Grant Program Competition File prior to the initiation of application reviews for discretionary grant program competitions. OESE Comments OESE stated that it recognizes, and does not dispute, that there was an issue in the peer reviewer selection process regarding the timeliness of its check of potential reviewers to ensure that they were not on the debarment and suspension list. However, OESE believed the issue was discovered at a “… point [at which] it could be easily addressed… .” OESE also expressed its belief that the issue was minor, as evidenced by the fact that no reviewers were actually found on the debarment and suspension list, and therefore should not have risen to the level of a finding. OESE described the issue as a “… harmless, procedural matter that was conducted a little late and that was corrected by the Department on a timely basis, as soon as it was discovered, and did not in any way affect the integrity or quality of the competition or the review process.” OIG Response While we acknowledge that no RTT peer reviewers were found on the debarment and suspension list, we do not concur with OESE’s assertion that it was discovered at a “… point [at which] it could be easily addressed,” nor its characterization of the issue as a “… harmless, procedural matter... .” The 16 RTT finalists had already been publicly announced at the time of the debarment and suspension list check, meaning that the integrity of the review process could have been compromised had any reviewers subsequently been found on the list. The fact that no 8 Overall, 49 of the 58 selected peer reviewers actually reviewed RTT Phase 1 applications. Final Audit Report Page 8 of 8 ED-OIG/A19K0006 reviewers were identified in the EPLS when the check was conducted does not obviate the need to complete the review earlier in the selection process or lessen its significance as an internal control procedure. SCOPE AND METHODOLOGY To accomplish our objectives, we performed a review of internal control applicable to the Department’s process for screening and selecting peer reviewers for discretionary grant program competitions. This included reviews of applicable Department policies and procedures; prior audit reports from OIG and other agencies, through which we sought to identify any potential vulnerabilities in this area; and the Government Accountability Office’s Standards for Internal Control in the Federal Government. We also gained an understanding of procedures specific to the screening and selection of RTT peer reviewers. We reviewed the authorizing legislation and program regulations, as well as various materials provided by Department officials and/or posted on the Department’s ARRA website. We conducted interviews with these same officials in order to obtain additional information on the reviewer selection process. We reviewed the RTT Grant Competition File and ATRP to determine whether they contained all required information. We reviewed all 58 selected peer reviewers’ files for completeness, reviewed and assessed the appropriateness of the 15 waivers granted by the Department, and conducted independent searches to confirm the accuracy of reported information and identify any discrepancies or other matters of concern. This included verification of eligibility based on an individual’s presence in the EPLS and a review of Federal Election Commission data to establish whether it might appear that political affiliation played a role in the selection process. We also reviewed documentation maintained for a random sample of 25 of the 130 individuals (19 percent) who were eliminated from consideration following the initial conflict of interest questionnaire and for all 16 individuals who were eliminated from consideration following the final conflict of interest questionnaire. This was done to determine the reasoning behind such decisions and gain assurance that the Department did not arbitrarily eliminate prospective reviewers from the pool of applicants. We relied on computer-processed data obtained from OESE staff to identify our universe of non- selected applicants. There were no other data sources readily available to independently corroborate the completeness of this data. However, the data were deemed sufficiently reliable for the purpose of this audit and adequately support resulting conclusions and recommendations. The scope of our review was limited to the Department’s screening and selection of peer reviewers for Phase 1 of the RTT discretionary grant program competition. We conducted fieldwork at Department offices in Washington, D.C., during the period March 2010 through April 2010. We provided our audit results to Department officials during an exit conference held on April 28, 2010. Our audit was performed in accordance with generally accepted government auditing standards appropriate to the scope of the review. The standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on the audit objectives. Anyone knowing of fraud, waste, or abuse involving U.S. Department of Education funds or programs should call, write, or e-mail the Office of Inspector General. Call toll-free: The Inspector General Hotline 1-800-MISUSED (1-800-647-8733) Or write: Inspector General Hotline U.S. Department of Education Office of Inspector General 400 Maryland Ave. S.W. Washington, DC 20202 Or e-mail: firstname.lastname@example.org Your report may be made anonymously or in confidence. For information on identity theft prevention for students and schools, visit the Office of Inspector General Identity Theft Web site at: www.ed.gov/misused The Department of Education’s mission is to promote student achievement and preparation for global competitiveness by fostering educational excellence and ensuring equal access. www.ed.gov Attachment 1 Overview of the Department’s RTT Peer Reviewer Screening and Selection Process OESE’s ATRP included all required information and provided a comprehensive overview of the RTT peer reviewer selection process. The Department employed a 10-step selection process which involved no less than 35 Department employees and included at least 4 separate checks for conflicts of interest, performed by personnel from 3 distinct entities – OESE, OGC, and a private contractor.9 Applicants were screened for both direct and indirect conflicts of interest,10 and in instances where such conflicts were identified, either dropped from consideration and informed of the reason(s) or granted a waiver to participate in the RTT Phase 1 application review process. These screenings were conducted by the entities described above, both singularly and in consultation with each other. OESE staff initially identified the RTT peer reviewer applicants that possessed the necessary expertise and experience, as outlined by the Secretary in his August 2009 request for reviewers. These applicants were asked to complete a questionnaire designed to identify direct conflicts of interest. Responses were provided to OESE and entered into a database to allow review by program staff. Prospective candidates with a direct conflict of interest were identified and removed from consideration. Subsequently, a team of Department officials consisting of career employees and political appointees with knowledge and experience concerning RTT program goals ranked the remaining candidates on the basis of their qualifications to serve.11 This process identified 74 preliminary finalists who were asked to complete another questionnaire designed to identify indirect conflicts of interest. The preliminary finalists were also asked whether they were aware of any other circumstances that might cause someone to question their ability or objectivity. This item was intended to raise awareness of any actual, potential, or perceived conflicts of interest. Responses were provided online through a survey tool administered by a private contractor. In mid-December 2009, OESE provided the contractor with the preliminary finalists’ responses to the questionnaire and asked that it run targeted internet searches on each individual to identify any issues or affiliations not previously disclosed. The contractor conducted follow-up phone calls with those for whom clarification and/or additional information was needed and documented their responses in a conflict of interest report prepared for each candidate. The reports were subsequently provided to officials in OGC’s Ethics Division, who in consultation with OESE, reviewed all available information, contacted some individuals again by phone to clarify their responses, and made the final decision on whether a prospective reviewer would be 9 See footnote 2 on page 3. 10 See footnote 3 on page 3. 11 The Department sought applicants with expertise in the following four areas: 1) education policy, 2) education reform, 3) capacity and scale, and 4) application review and evaluation. It was also suggested that applicants possess a broad understanding of each of the four education reform areas embodied by RTT: 1) standards and assessments, 2) teachers and school leaders, 3) data systems, and 4) school turnaround. cleared to serve. In the end, the Department invited 58 individuals to participate in the RTT Phase 1 peer review, either as reviewers or alternates.12 During this process, the Department deemed some applicants to have an indirect conflict of interest for which a waiver could be granted. Specifically, OGC recommended waivers for 21 of the 58 individuals (36 percent) invited to serve as RTT Phase 1 peer reviewers. However, 5 of these 21 individuals served as alternates, meaning they did not review any applications and therefore, according to Department officials, did not require waivers unless called upon to do so. In addition, one individual who would have required a waiver based on a prior affiliation with one State Education Agency did not receive a waiver because the applicable State did not submit an RTT Phase 1 application. We subsequently located approved waivers on file for the 15 individuals who reviewed RTT Phase 1 applications and required waivers. None of the alternates who would have required waivers were asked to serve. Upon selection, reviewers and alternates were required to sign a reviewer agreement, thereby certifying to the completeness and accuracy of information provided by them during the conflict of interest review and acknowledging the need for integrity in the review process and confidentiality with respect to their role as a peer reviewer. Individuals were also advised to promptly notify the appropriate program official if they became aware of any circumstances that might cause someone to question their impartiality or if they discovered that they may have a previously unidentified direct or indirect conflict of interest while reviewing the applications assigned to them. Department officials also stated that certain aspects of the design of the RTT competition itself helped mitigate the possibility that any individual peer reviewer may not be impartial. These included the following: Reviewers were not permitted to serve on a panel reviewing the application submitted by the State in which he or she resides. Reviewers granted a waiver for an indirect conflict of interest were not permitted to review an application from the State(s) that gave rise to the conflict. Applications were randomly assigned to reviewers after the States for which reviewers had waivers or disqualifications were “blacked out.” Each application was reviewed by five reviewers. Both before and during the review process, program officials and reviewers were reminded that throughout the course of the review, they must identify any circumstances that might cause a reasonable person to question a reviewer’s impartiality in serving. 12 Thirteen of these 58 individuals (22 percent) were initially designated as alternates. Five alternates were subsequently included in the RTT Phase 1 application review. Attachment 2 Summary of Participants and Outcomes of Each Step of the Department’s RTT Peer Reviewer Screening and Selection Process Attachment 3 Acronyms/Abbreviations Used in this Report ARRA American Recovery and Reinvestment Act of 2009 ATRP Application Technical Review Plan Department U.S. Department of Education EPLS Excluded Parties Listing System GSA Government Services Administration OESE Office of Elementary and Secondary Education OGC Office of the General Counsel OIG Office of Inspector General RTT Race to the Top Secretary U.S. Secretary of Education Attachment 4 UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF ELEMENTARY AND SECONDARY EDUCATION TO: Ms. Michele Weaver‐Dugan, Director Operations Internal Audit Team FROM: Joseph Conaty, Director Academic Achievement and Teacher Quality Programs SUBJECT: Draft Audit Report: The Department’s Process for Screening and Selecting Peer Reviewers for the Race to the Top Grant Program, A19‐K0006 We appreciate the opportunity your office provided for a meeting to discuss the possible findings from the audit work on "The Department's Process for Screening and Selecting Peer Reviewers for the Race to the Top Grant Program," and the opportunity to provide comments on the draft report. We found the audit team working on this matter to be open, objective, cooperative, and highly professional. As a general matter, we appreciate that the Office of Inspector General recognizes the many careful steps we included in our selection and review process of potential peer reviewers to ensure fairness, objectivity, and highly skilled group of peer reviewers. The process was very effective in meeting its goals and worked exceedingly well. At the same time, we recognize that there was one minor issue in the competition regarding checking potential reviewers to ensure that they were not on the debarment and suspension list. While we do not dispute that ideally this check should have occurred on a more timely basis, we do not believe that this issue should rise to the level of a finding and have such prominence in this report. When the Department discovered it, it was a point that it could be easily addressed, and Department officials immediately addressed this issue and, in fact, no reviewers were on the debarment and suspension list. At most, therefore, this was a harmless, procedural matter that was conducted a little late and that was corrected by the Department on a timely basis, as soon as it was discovered, and did not in any way affect the integrity or quality of the competition or the review process. We appreciate the overall positive nature of the report in identifying the overall very fair and successful process, and thank you for the opportunity to comment. Please let us know if you have any questions. Thanks.
The Department's Process for Screening and Selecting Peer Reviewers for the Race to the Top Grant Program.
Published by the Department of Education, Office of Inspector General on 2010-08-16.
Below is a raw (and likely hideous) rendition of the original report. (PDF)