UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF THE INSPECTOR GENERAL FEB 1 6 2005 Sally L. Stroup Assistant Secretary Office of Postsecondary Education 1990 K Street NW Room 7115 Washington, DC 20006 Dear Ms. Stroup: This final audit report, Control Number ED-OIG/ A07-E0009, presents the results of our audit of the Talent Search program at the U.S. Department of Education (Department). The objective of our audit was to detennine, through audits of a nationwide sample of Talent Search projects, whether the practice of overstating the funded target population is widespread. This report is a compilation of work conducted at the Federal TRIO Office (TRIO Office), as well as issues identified from six Talent Search project audits listed in Attachment 1. We determined that the TRIO Office did not maintain sufficient internal control over Talent Search participant numbers because it did not (i) properly maintain the records and procedures needed to readily determine the correct number of participants planned or (ii) provide the monitoring and policy guidance needed to insure accurate reporting of participants served. We conclude that the practice of overstating the funded target l population could be widespread. As a result, the Department may be using overstated Talent Search participant numbers for assessing grant performances and reporting to Congress and the general public. We recommend changes to the grant award process and improvements in record-keeping as well as enhanced monitoring and policy guidance to improve the accuracy of Talent Search participant numbers. We provided the Department with the draft of this report on November 2, 2004. In its response dated January 3, 2005, the Department indicated that action has been taken to address each of the recommendations (response attached in full in Attachment 2). Based on the Department's response, minor edits were made to the finding. We also provided more information on the sample selection methodology and results. I Due to the small number of projects audited, we cannot combine the results of our audits to reliabJy estimate the totaJ overstatement of Talent Search participants reported as planned or served. However, we are 90 percent confident that participant numbers were overstated in some amount for reported participants planned (in at least 20 percent of projects) and participants served (in at least 68 percent of the projects). 400 MARYLAND AVE .. s.w.. WASHINGTON. DC 20202-1510 ",w",.ed.go.· Our mission is /0 ensure equal access to education and to promote educatiorwi exceUence throughout the nation. AUDIT RESULTS Talent Search Participants, Both Planned And Served, Were Overstated Planned Talent Search participants posted on the TRIO website were overstated for half of the projects audited and Talent Search participants served were over-reported for all the projects audited. Differences in the Talent Search participant numbers, both planned and served, were noted from various Department sources for the six audited projects. These sources are the basis for the participant numbers the Department may use for monitoring, generating reports, and providing information to Congress and the general public. Participants planned were overstated due in part to the reduction of initial numbers through negotiations between the Department and applicants. The differences in the stated number of planned participants often occurred because the reduced numbers were not reflected on the Department's internal database or the official TRIO website. In addition, poor maintenance of official grant files further contributed to the inconsistent numbers. Participants served were overstated because grantees often reported participants for whom they could not document eligibility and/or at least one eligible service. Differences In Talent Search Participant Numbers From Various Department Sources 2001–2002 Award Year PLANNED SERVED Grant TRIO Actual Grantee Application Website Negotiated Reported Projected * Luther College 900 850 625 610 363 University of New Hampshire 1,350 1,200 1,150 1,207 1,089 Case Western Reserve University 800 600 600 605 399 Wahupa Educational Services 2,665 2,300 2,300 2,584 2,381 Communities in Schools of San 700 600 600 604 481 Antonio LULAC National Educational 20,300 14,700 12,200 15,228 13,608 Service Centers, Inc. *The number of participants served are point estimates based on our statistical samples. Participants Planned Were Routinely Reduced and Grant Records Were Not Well Maintained. According to TRIO officials, Talent Search applicants often proposed larger projects than could be realistically funded. For the six projects audited, the Department reduced the ultimate award amount, which resulted in a lower participant number. Once projects were selected, TRIO officials developed an initial list of all funded Talent Search projects for the award period. This funded list was used as the basis for the TRIO Office internal management database and posted on the Talent Search website. However, prior to the 2003-2004 award year, it was common practice for the Department to continue to negotiate the participant number through a partnership agreement. The partnership agreement often contained another revised participant number, but the database and website were not always updated with the revised number. ED-OIG A07-E0009 Page 2 The handbook for discretionary grants2 requires program staff to create and maintain an official grant file for each application selected for funding. Program specialists are also responsible for updating participant numbers by reporting all changes of information to the Program Management and Development Team Leader. However, discussions with program staff indicated that file maintenance is a low priority and TRIO Office management has not developed specific guidance for program specialists to assure timely and accurate updates to the website. For three of the six projects audited, the TRIO Office had difficulty determining the exact number of participants these projects planned to serve because the grant file for one project was missing, and proposed revisions to the 1998–1999 partnership agreements were incomplete for all three. In addition, we found three grantees had further reductions that had not been updated on the TRIO website. Differences in Talent Search Participants Planned 2001-2002 Award Year Grant TRIO Actual Grantee Application Website Negotiation Overstated Luther College1 900 850 625 225 University of New Hampshire 1,350 1,200 1,150 50 Case Western Reserve University 800 600 600 0 Wahupa Educational Services2 2,665 2,300 2,300 0 Communities in Schools of San Antonio 700 600 600 0 LULAC National Educational Service 20,300 14,700 12,200 2,400 Centers, Inc.3 Notes: 1. Partnership agreement in file was 850, however in 1999 the grantee requested to reduce the participant number to 625. The program specialist sent a letter dated September 4, 2002, while we were on site doing audit work, approving the 625 participants retroactively to 1998. 2. Missing File, grantee did not have a partnership agreement. 3. Approved participant number change not in grant file, but obtained from auditee. It may be difficult for the TRIO Office to determine the agreed upon numbers of Talent Search participants funded for monitoring or reporting purposes, since neither the award database nor the official grant files were complete for the six audited projects. In addition, during the course of our audit work, we found no evidence that the Department had conducted on-site monitoring of the six individual projects. The Federal TRIO Director told us that the uncertainty about the number of participants funded no longer exists, because partnership agreements were discontinued in the 2003- 2004 award year, and grantees will be held to the original grant application. Grantees received a letter informing them of this action. The letter requested a statement declaring the number of participants to be served in 2003-2004; however, it was unclear whether that number needed to be approved by a program specialist. To test the effect of the discontinuation of the partnership agreements, we randomly selected 89 of the 471 2003- 2 Handbook for the Discretionary Grant Process, Handbook OCFO-04, § 5.12. ED-OIG A07-E0009 Page 3 2004 Talent Search projects. Although one grant file was missing, only 52 of the remaining 88 files contained a statement on the number of participants the grantee would serve.3 Again we noted differences from the grant application and differences in the website list, with no documentation supporting the change. We noted some program specialists stamped approved on the participant statement, but for statements missing from the program file, or having no notation, the number of participants funded remains questionable. Management’s change in policy implemented during the 2003-2004 grant award period did not establish an effective internal control to provide certainty as to the number of Talent Search participants funded. Participants Served Were Overstated by Grantees. None of the six grantees audited could provide sufficient reliable documentation to support the number of participants reported to the Department as served for the 2001-2002 award period. Talent Search Participants Served From Audits of 2001-2002 Grant Projects Overstated Grantee Reported Served * Number % Luther College 610 363 247 40 % University of New Hampshire 1,207 1,089 118 10 % Case Western Reserve University 605 399 206 34 % Wahupa Educational Services 2,584 2,381 203 8% Communities in Schools of San Antonio 604 481 120 20 % LULAC National Educational Service Centers, Inc. 15,228 13,608 1,620 11 % *The numbers served are point estimates based on audit statistical samples of reported participants. In part, the lack of quality data is due to the Department not providing clear, concise, and consistent guidance necessary to ensure that grantees understand and comply with Federal regulations. Consequently, grantees reported ineligible project services; undocumented project services; undocumented citizenship; and served non-participants, prior to determination of participant eligibility. Additional Departmental guidance suggested by grantee officials included • sample forms, • a data library for directors, • examples of best practices, • clarification of allowable and unallowable project services, and • clarification of adequate documentation for eligibility, i.e., project services, citizenship, and first generation status. The following are two examples of inconsistent guidance the Department provided to a grantee and the OIG. 3 Although files were incomplete, program specialists were able to locate statements for many grants. ED-OIG A 07-E0009 Page 4 • One grantee stated that it began serving students who were neither citizens nor permanent residents after attending a presentation where a TRIO Educational Specialist at a TRIO Conference said “intent” was not defined in the regulation and was “open to interpretation by the individual.” The grantee concluded that under its interpretation, “intent” was simply getting a family’s home address or showing that the student was enrolled in a public school. However, the regulation states that an individual is eligible to participate in a Talent Search project if the individual is in the United States for other than a temporary purpose and provides evidence from the Immigration and Naturalization Service (INS) of his or her intent to become a permanent resident (34 C.F.R. § 643.3(a)). The regulation clearly states that evidence must be provided from the INS to show the individual’s intent to become a permanent resident. • In February 2003, during one of our Talent Search audits, we sought official guidance from the TRIO Office on accepting information on first generation status from a minor child. A TRIO official provided a written response stating that first generation status should be verified by the parent or other knowledgeable adult. In March 2004, while we were continuing our audit work, the Federal TRIO Director informed us that the Department had reversed its position and would now accept information provided by the minor with no further verification. The Department’s Participant Numbers May Be Overstated in Its Reports to Congress and the General Public. The TRIO Office uses Talent Search participant numbers to monitor the program, generate reports, and provide information to Congress and the general public. Based on the results of our work, these participant numbers may be overstated. Unless the TRIO Office provides consistent and reliable guidance to grantees, and makes monitoring a priority, overstated participant numbers may continue. Participant numbers are reported both within and outside the Department: • To respond to Congressional requests. • To produce a TRIO profile report. • To assess a grantee’s progress in meeting its approved goals and objectives. • To determine if a grantee should receive prior experience points for a new grant or continuation of an existing grant. • To provide national information on project participants and program outcomes, i.e., Council for Opportunity in Education (COE), TRIO website. • To calculate the average cost per participant. • To report information on the Program Assessment Rating Tool (PART). Recommendations: We recommend that the Assistant Secretary of the Office of Postsecondary Education 1.1 Modify the application process by ED-OIG A 07-E0009 Page 5 • requiring grantees to propose realistic participant numbers in the grant application, and • discontinuing the practice of revising planned participant numbers after the Talent Search grant has been awarded. 1.2 Ensure that program staff follow policies and procedures in • maintaining official grant files in accordance with internal guidelines, • prescribing specific procedures for updating grant information to senior program staff, and • documenting justification for any deviation from the approved grant application, including a change in planned participant numbers. 1.3 Make monitoring a higher priority by conducting on-site reviews focusing on grantees that • are designated high-risk, • have not submitted required reports, i.e., single audit, annual performance reports, or • have not met program objectives, including serving the funded participant number. 1.4 Establish a mechanism for publishing specific, accessible, and consistent policy guidance that enables grantees to effectively administer their project. Guidance may include examples of forms and best practices, and should address important questions such as: • What is an acceptable project service? • When do you count an individual as a participant? and • What is sufficient reliable documentation to establish individual participant eligibility? Auditee Response The Department’s response noted that, upon Office of Postsecondary Education (OPE) and the Office of Chief Financial Officer review, some of the OIG’s recommendations, contained in the six audits this report was based on, were modified or not sustained and four have been resolved. In addition, OPE expressed an overall concern with the methodology used to demonstrate a widespread programmatic practice because the audits did not constitute either a true random sample or a sample with the statistical power to extrapolate to the entire population of grantees. OPE stated that it was improper to combine a purposively selected largest project with a random sample of other projects and that a proper “nationwide sample” should be selected by grouping auditees into size categories for sample selection. OPE reported that, based on the six issued audits, the Federal TRIO Director had already implemented policy and procedural guidance to ensure that grantees do not overstate ED-OIG A 07-E0009 Page 6 participant populations, but acknowledged that there were some weaknesses in implementing the changes. OPE is establishing the following additional policy and procedural improvements in response to the specific recommendations. Responses to Specific Recommendations 1.1 Talent Search applicants will be required to propose realistic participant numbers in their grant applications and be required to meet the proposed goals and objectives based on funding guidelines to be published. In Spring 2004, a letter to current grantees informed them that they will be held to participant numbers previously established. 1.2 Steps have already been taken that would ensure program staff maintain official grant files in accordance with internal guidelines including staff training and inclusion in revised EDPAS agreements. In addition program staff must obtain approval from the Team Leader to reduce a grantees participant numbers. If approval is granted, it is the program specialist’s responsibility to document the action in the grant file and ensure that the database is updated. 1.3 The Department pointed out that monitoring has and will continue to be a high priority for the TRIO Program Office and there are currently no grantees designated as “high risk.” 1.4 The Department responded that the TRIO program office has several mechanisms in place to disseminate information to the TRIO community, such as the Training Program for Federal TRIO Programs, a recently established TRIO newsletter, and the TRIO Website. OIG Comments While the Department either modified or did not sustain some of the recommendations for the four resolved audit reports, it did concur that the Talent Search participant numbers reported by the grantees to the Department were not fully supported, and therefore overstated. Of the four audits resolved, the Department fully sustained the findings and recommendations for two audits. For the third, the Department fully sustained the finding that participants were over-reported, but did not sustain the monetary refund because the entity provided additional post-audit documentation. The Department partially sustained the number of over-reported participants on the fourth, yet chose to assess a pro-rata monetary recovery even though it agreed that the reported participants were overstated by approximately 11 percent, and that the project failed to meet the minimum program size by that same percentage. We disagree that the sample did not represent a “nationwide sample” and that the results did not support our audit objective. Our sample, representative of all Talent Search projects, was randomly selected from a stratified universe of all funded Talent Search projects for the 2001-2002 award year, just as suggested in OPE’s response. The largest ED-OIG A07-E0009 Page 7 project was not selected purposively, but was selected because it was the only project in its stratum. A chart showing the detail of each of the strata has been added to the Audit Objective, Scope, and Methodology section of this report (page 10). To further explain the audit results, we have added a footnote to page 1 of this report to explain, that while the sample was too small to project the total number of participants overstated (a variable sample), it was sufficient to conclude that the existence of projects with overstated participant numbers (an attribute sample) was widespread. Specific recommendations 1.1 We agree with the action taken by the Department. 1.2 We agree with the action taken by the Department. 1.3 We commend the TRIO Program Office for the number of TRIO site visits conducted over the last two years, and for providing program staff the necessary training to conduct on-site monitoring and fiscal reviews. However, it is not clear how many site visits were conducted at Talent Search projects. Monitoring and on-site reviews of Talent Search projects is crucial to ensure that the information submitted by grantees is complete and reliable. As shown in this report, four of the six projects audited failed to meet their funded participant number, and three of the six projects failed to serve the required minimum of 600 participants. Therefore, the Department should continue to make monitoring and on-site reviews a priority focusing on grantees that have not met their program objectives, which includes serving the funded participant number. 1.4 The Department listed five priorities to be covered through its Training Program for Federal TRIO Programs. However, the Department does not have control over what particular topics or materials the grantees of the Training Program offer the TRIO community. Further, we noted the TRIO newsletters (Fall and Winter) and the TRIO Website contained general information about the TRIO programs. We did not find information in relation to the findings or recommendations from the OIG Talent Search audits, the Department’s program site reviews, or suggestions made by grantees, such as providing examples of best practices, a data library for directors, or sample forms. Providing specific information to grantees may help reduce or eliminate these types of findings in the future. OTHER MATTERS Talent Search Participant Numbers Fall Below the Minimum 600 Participant Requirement According to 34 C.F.R. § 643.32(b), Talent Search grantees are required to serve a minimum of 600 participants each budget period. Three of the six grantees failed to serve 600 participants for the budget period reviewed. If the Department is not ED-OIG A07-E0009 Page 8 monitoring participant numbers and is not enforcing the regulatory requirement that grantees meet the current minimum participant number, then the regulation is meaningless and the Office of Postsecondary Education (OPE) should consider a regulatory change to establish a more appropriate minimum number of participants required for an eligible Talent Search project. Talent Search Projects Provide Services to Uncounted and Often Ineligible Students In two of our six audits, we identified instances where services were routinely provided simultaneously with the collection of Intake forms (participant application) without verifying information or obtaining necessary documentation for participant eligibility, resulting in services being provided to ineligible individuals. OPE should consider a regulatory change to require grantees to determine eligibility prior to providing services to individuals. This would avoid diluting the services to eligible individuals. BACKGROUND Title IV of the Higher Education Act of 1965, as amended (20 U.S.C. § 1070a-11 and 12), authorizes the Talent Search program. The Talent Search program is governed by the regulations in 34 C.F.R. Part 643. All regulatory citations in the report are to the regulations in effect as of July 1, 2001, for our work on the six individual audits, and July 1, 2003, for our work at the Federal TRIO office. In 1965, Talent Search was created as part of the Higher Education Act. It is one of nine TRIO outreach and support programs. The goal of Talent Search is to increase the number of youths from disadvantaged backgrounds who complete high school and enroll in postsecondary education. The Talent Search program identifies and assists individuals from disadvantaged backgrounds who have the potential to succeed in higher education. The program provides academic, career, and financial counseling to its participants and encourages them to graduate from high school and continue on to the postsecondary institution of their choice. Talent Search also serves high school dropouts by encouraging them to re-enter the educational system and complete their education. Talent Search projects may be sponsored by institutions of higher education, public and private agencies or organizations, a combination of these, and in exceptional cases, secondary schools. Students must be between the ages of 11 and 27 and have completed the fifth grade. In any given project, at least two-thirds of the participants must be low- income individuals and potential first-generation college students. A veteran, regardless of age, may participate if otherwise eligible. The Department awarded 360 Talent Search projects in the 2001-2002 award year with appropriations totaling $109,960,406. In the 2003-2004 award year, the Department reports 471 Talent Search projects were awarded with appropriations totaling $144,810,906. ED-OIG A07-E0009 Page 9 AUDIT OBJECTIVE, SCOPE, AND METHODOLOGY The objective of our audit was to determine, through audits of a nationwide sample of Talent Search projects, whether the practice of overstating the funded target population is widespread. Our review focused on the Department’s procedures to assure the accuracy of participant numbers. We considered the results of each of the six 2001-2002 individual audits. (See Attachment 1.) In addition, we randomly selected 89 of 471 2003-2004 Talent Search grant files for review. To achieve our overall audit objective we: • reviewed applicable laws and regulations governing the Talent Search program; • conducted interviews with program officials and staff in the Talent Search office located in Washington, DC; • reviewed a sample of the Department’s current Talent Search grant files; • obtained and analyzed documentation related to Talent Search projects; and • audited six Talent Search projects. We established four strata for the 360 Talent Search projects from the TRIO website based on the number of participants reported ranging from 480 to 14,700. We selected the largest project (in its own stratum) and randomly selected the remaining five projects from the remaining three strata. Number of Projects Number of Population (Based on Talent Projects Selected Count Search Website) from each Strata for Sample 14,700 1 1 1,001 – 2,500 64 2 700 – 1,000 166 1 < 700 129 2 The primary objective for each of the individual audits was to determine, through a random sample of participants from each project, if each grantee had administered its Talent Search grant in accordance with the law and Talent Search regulations governing the documentation of participant eligibility (34 C.F.R. § 643.32). A secondary objective was to determine if each of the individual grantees met the two-thirds assurance requirement for serving low-income and first-generation participants. The Office of Inspector General (OIG) issued individual audit reports to each Talent Search grantee. (See Attachment 1 for details.) We relied on computerized records to determine the number of Talent Search projects funded for the 2003-2004 award year. Based on our assessments, we concluded that the data used was sufficiently reliable for the purpose of our audit. ED-OIG A 07-E0009 Page 10 An entrance conference was held at the Federal TRIO Office located in Washington, DC on November 19, 2003. Fieldwork was conducted during the weeks of November 17-21, 2003, and May 3-7, 2004. An exit conference was held on September 28, 2004, with Office of Postsecondary Education officials, including Federal Talent Search officials. This audit and each of the six individual audits were conducted in accordance with generally accepted government auditing standards appropriate to the scope of each audit. STATEMENT ON INTERNAL CONTROLS As part of our review we assessed the system of internal controls, policies, procedures, and practices applicable to the Department’s administration of the Talent Search program relevant to the scope of our review. Our assessment included a determination of whether the Federal TRIO Office provided a reasonable level of assurance that the number of Talent Search program participants funded and served is being appropriately recorded and reported. For the purpose of this report, we assessed and classified the significant controls into the following categories: Participant numbers funded during the 2001-2002 and 2003-2004 award periods; and Participant numbers reported as served during the 2001-2002 award period. Because of inherent limitations, a study and evaluation made for the limited purpose described above would not necessarily disclose all material weaknesses in internal control. However, our assessment disclosed significant internal control weaknesses, which adversely affected the Talent Search program office’s ability to record and report accurate Talent Search participant numbers for the periods of our review. These weaknesses and their effects are fully discussed in the AUDIT RESULTS section of this report. ADMINISTRATIVE MATTERS Corrective actions proposed and implemented by your office will be monitored and tracked through the Department’s Audit Accountability and Resolution Tracking System (AARTS). Department policy requires that you develop a final corrective action plan (CAP) for our review in the automated system within 30 days of the issuance of this report. The CAP should set forth the specific action items, and targeted completion dates, necessary to implement final corrective actions on the findings and recommendations contained in this final audit report. In accordance with the Inspector General Act of 1978, as amended, the Office of Inspector General is required to report to Congress twice a year on the audits that remain unresolved after six months from the date of issuance. ED-OIG A07-E0009 Page 11 In accordance with the Freedom of Information Act (5 U.S.c. § 552), reports issued by the Office of Inspector General are available to members of the press and general public to the extent infonnation contained therein is not subject to exemptions in the Act. We appreciate the cooperation given us during this review. If you have any questions, please call Janice Keeney, Assistant Regional Inspector General for Audit, at 816~268 ~ 0500 or Richard J. Dowd, Regional Inspector General for Audit, at 312-886-6503. Sincerely, J(e~W Helen Lew Assistant Inspector General for Audit Services Attachments ED-OIG A07-E0009 Page 12 Attachment 1 OIG REPORTS FOR TALENT SEARCH PROJECTS AUDITED WITH SUMMARY OF AUDIT FINDING ISSUES Audit Auditee OIG Final Findings Control Report Number Issued A07- Luther College March 28, a, b, d, e C0031 http://www.ed.gov/about/offices/list/oig/auditreports/a07c0031.pdf 2003 A07- University of New Hampshire January 15, a, d, e D0001 http://www.ed.gov/about/offices/list/oig/auditreports/a07d0001.pdf 2004 A07- Case Western Reserve University July 11, 2003 a, b, c, d, D0002 http://www.ed.gov/about/offices/list/oig/auditreports/a07d0002.pdf e, f, i A07- Wahupa Educational Services November a, c, d, e, h D0009 http://www.ed.gov/about/offices/list/oig/auditreports/a07d0009.pdf 25, 2003 A07- Communities in Schools of San Antonio January 29, a, b, c, d, e D0015 http://www.ed.gov/about/offices/list/oig/auditreports/a07d0015.pdf 2004 A07- LULAC National Educational Service Centers, Inc. June 18, a, c, d, e, g D0024 http://www.ed.gov/about/offices/list/oig/auditreports/a07d0024.pdf 2004 Legend – Findings a: Reported participant numbers were overstated. b: Did not meet the minimum (600) participant number. c: Did not adequately document citizenship status, i.e., ineligible participant. d: Did not adequately document project services. e: Counted ineligible project services, i.e., mailings, newsletters, administrative activity. f: Prior period participant – no services in 2001–2002. g: Counted duplicate participants. h: Did not meet two-thirds assurance requirement. i: No written policies and procedures. ED-OIG A07-E0009 Page 13 UNITED STATES DEPARTMENT OF EDUCATION OFFICE OF POSTSECONDARY EDUCATION Attachment 2 THE ASSISTANT SECRETARY MEMORANDUM DATE: By TO: Richard J. Dowd Regional Inspector General for Audit FROM: Sally L. StrouIiJ) SUBJECT: Comments on Draft Audit Report of the Talent Search Program (Control Number ED-OIG /A07-E0009) We have reviewed the draft audit report of the Talent Search Program at the U.S. Department of Education (Department). The primary objective of the audit was to determine, through audits of a nationwide sample of Talent Search projects, whether the practice of overstating the target population was widespread . This draft audit report is based on six audits of 200 \-02 Talent Search grants, the first of which was completed in April 0[2003 and the last in June 0[2004. The Office of Postsecondary Education has previously responded to the findings in each of these audits. It is important to note that, upon OPE and the Office of Chief Financial Officer review, some of the Office of Inspector General's (OIG's) recommendations were modified or not sustained. In fact, four of the audits have already been resolved . Further, based on the information the OIG previously provided to OPE, some corrective actions have already been taken on a program-wide scale and further program improvements are planned. While OPE's comments to the individual findings and recommendation of this draft audit report are provided below, we wanted to bring to your attention our overall concern with the methodology used for the stated objective of this audit. The OIG has attempted to determine whether a programmatic practice is widespread using a group of audits that do not constitute either a true random sample or a sample with the statistical power to extrapolate to the entire population of grantees. The report states that "a nationwide sample" was used. This wording does not provide any statistical information and will likely be misleading to the general population, in that it implies there is validity in applying the sample to the national program. Actually, given the methodology used, the audit objective is not attainable. In general, it appears that the determinations in this audit provide little value, ifany, beyond what has already been achieved through the original six audit reports. 1990 K STREET, N.W. WASHINGTON, D.C. 20006 Our mission is to enslJ.re eqlUl! access to edlJ.cntion and to promote educntional excellence throughout die Nation. Page 2 - Richard J. Dowd If the OIG intends to publish the audit anyway, it should expand upon stated limitations of the methodology and do this at the very beginning of the report. The draft audit report now states that "Due to the small number of projects audited, we cannot combine the results of our audits to reliably estimate the total overstatement of Talent Search participants reported as planned or served." We believe that the reader should be advised of the following: • The six audits do not constitute a sampling that is reflective of the programs' grantees as a whole. The OIG audited the largest Talent Search project, along with 5 other randomly selected projects from the 360 projects funded in 2001-02 (less than 2 percent of the Talent Search projects). It is not appropriate to combine the findings from the largest Talent Search project with the findings from the 5 randomly selected projects. The largest project was purposively selected, while the others are a random sample. A truly random sample cannot include any purposively selected projects. • Talent Search projects vary considerably in size. A sample of 5 does not provide the statistical power to allow one to extrapolate to the entire universe of projects. A stronger sampling methodology would be to carefully examine the universe of projects and group them into appropriate size categories. For example, if a large proportion of projects serve 1,000-5,000 participants, that would be one size category from which to select a sample of projects to audit. Given the weaknesses with this audit methodology, OPE strongly questions the validity of including the section of this draft report entitled: "The Department's Numbers May Be Overstated in Its Reports to Congress and the General Public." This section provides a listing of uses for the TRIO information reported. Given that the OIG has not clearly identified and documented that there is currently any program-wide abuse, the logic for this section is questionable. FINDING NO. ]: Talent Search Participants, Both Planned And Served, \Vere Overstated The Office of the Inspector General (OIG) states that planned Talent Search participants numbers posted on the TRIO Web site were overstated for half(three) of the projects audited (a total of six) and that the number of Talent Search participants actually served were over-reported for all of the projects audited. The OIG further states that the TRJO Office did not maintain sufficient internal control over Talent Search participant numbers because it did not (1) properly maintain the records and procedures needed to readily determine the correct number of participants to be served, or (2) provide grantees with monitoring and policy guidance needed to insure accurate reporting of the number of participants served. The OIG concluded that the practice of overstating could be widespread. (For reasons stated above, this last statement is speCUlative rather than based on sound statistical methodology.) Page 3 - Richard J. Dowd Prior to the drafting of this audit report, the Federal TRIO Director had already implemented policy and procedural guidance to ensure that grantees do not overstate the participant population, as follows: • First, he discontinued partnership agreements in the 2003-2004 Award Year. Partnership agreements allowed the Department to continue to negotiate student participant numbers during the grant period. • Second, the Federal TRIO Director has established an internal process that now holds grantees accountable based on the number of participants specified in the original grant application. • Third, the grantee community has been informed of these changes. The OIG did identify some weaknesses in implementing these changes. However, it is not unusual for programmatic and procedural changes to encounter some problems in the first stage of implementation. The TRIO Program Office is currently establishing additional policy and procedural improvements to ensure that grantees provide accurate participant numbers. We elaborate further on these actions in our responses to specific recommendations. As a result of its audits, the OIG recommended changes to the grant award process and improvements in recordkeeping, as well as enhanced monitoring and policy guidance to improve the accuracy of Talent Search participant numbers . RECOMMENDATION 1.1 The OIG recommended that Assistant Secretary for the Office of Postsecondary Education modify the application process by: • Requiring grantees to propose realistic participant numbers in the grant application, and • Discontinuing the practice of revising planned participant numbers after the Talent Search grant has been awarded. RESPONSE 1.1 The TRIO Program Office will require Talent Search applicants to propose realistic participant numbers in their Talent Search grant application. The Department has already begun planning for the upcoming competition for grants with a projected application due date ofiate summer or early fall. Applicants will be required to meet the goals and objectives they propose in their application based on funding guidelines to be published in the Federal Register notice inviting applications. Current grantees will be informed that they will receive no more than a fixed percentage over their prior year's award amount. New applicants will be informed of the maximum award amounts and the minimum number of participants they must serve. Page 4 - Richard 1. Dowd In addition the TRIO Program Office has taken steps to ensure that current grantees meet their participant numbers. For example, in the spring of 2004, a letter was mailed to grantees with continuation grant awards infonning them that the project will be held to participant numbers previously established for the grant period. Also, the TRIO Program Office has implemented a policy that will deny Prior Experience points to grantees that have failed to meet their planned participant numbers. RECOMMENOAnON 1.2 The OIG recommended that Assistant Secretary for the Office of Postsecondary Education ensure that program staff follow policies and procedures in: • Maintaining official grant files in accordance with internal guidelines, • Prescribing specific procedures for updating grant infonnation to senior staff, and • Documenting justification for any deviation from the approved grant application, including a change in planned participant numbers. RESPONSE 1.2 The TRIO Program Office has already taken the following steps to ensure that program staff will maintain grant files in accordance with internal guidelines: • In the spring of 2004, the team responsible for administering the Talent Search Program was trained on the proper procedures for maintaining grant files. The training was based on the Grants Policy and Oversight (GPOS) Bulletin #2, "Organization of the Official File Folder." Each member of the staff was provided with a one·page handout titled, "Official Grant File Organization Guide." • The EDPAS agreements for staff have been revised to include language regarding grant file maintenance. The Team Leader and the Director have advised staff regarding the importance of keeping files up-to·date. It is anticipated that the e-Monitoring system will also be a tool for assisting staff in the efficient maintenance of project files. • The staff has been infonned that any request to reduce participant numbers must be approved by the Team Leader. The Team Leader will follow the Director 's guidance regarding grantee requests to revise participant numbers. If a grantee received approval to revise the participant numbers, the assigned program specialist must document the action in the grant file and ensure the database is updated. Page 5 - Richard J. Dowd RECOMMENDATION 1.3 The OIG recommended that Assistant Secretary for the Office of Postsecondary Education make monitoring a high priority by conducting on·site reviews focusing on grantees that: • Are designated high risk, • Have not submitted required reports, single audit, annual performance reports, or • Have not met program objectives, including serving the funded participant number. RESPONSE 1.3 Monitoring has been and will continue to be a high priority for the TRIO Program Office. In the last year, TRIO program staff and Program Oversight Staff (POS) conducted approximately 100 site visits. Many of the visits were to non.profit agencies that administer Talent Search projects. In the prior year, TRIO and POS staff conducted approximately 100 site visits of TRIO grantees. In addition, in November 2003, TRIO program staff participated in training for on·site monitoring and fiscal review techniques. OPE developed a reference manual for the fiscal review of grant projects and updated monitoring guides for several of the TRIO Programs including Talent Search. Currently, no Talent Search grantees are designated as "high risk" in accordance with the standards published in 34 CFR 74.14 of the Education Department General Administrative Regulations (EDGAR). However, the TRIO Program Office uses a variety of indicators to determine whether the Federal funds awarded to a particular grantee may be "at risk." The indicators include referrals from the Office of the Inspector General, complaints with merit, failure to submit required performance and other reports, "problem" audits, and poor performance. All relevant factors are considered in determining which grantees may require increased monitoring and/or a site visit. RECOMMENDAT ION 1.4 The OIG recommended that Assistant Secretary for the Office of Postsecondary Education estab lish a mechanism for publish ing specific, accessible, and consistent policy guidance that enables grantees to effectively administer their projects. Guidance may include examples of forms and best practices, and answer questions such as: • What is an acceptable project service? • When do you count an individual as a participant? Page 6 - Richard J. Dowd • What is sufficient reliable documentation to establish individual participant eligibility? RESPONSE 1.4 The TRIO Program Office already has several such mechanisms in place. One of the mechanisms for disseminating information to TRIO staff is through the Training Program for Federal TRIO Programs. This program provides training and policy guidance to staff and leadership personnel employed in or preparing for employment in projects funded under the Federal TRIO Programs. In fiscal year 2004, 13 new Training Program grants were awarded to train an estimated 3,688 staff in the following five areas: • Priority 1: Training to improve: budget management, recordkeeping and reporting student and project perfornlance and the evaluation of project perfornlance. • Priority 2: Training on: the legis lative and regulatory requirements for operat ion of the Federal TRIO programs; personnel management; and student aid. • Priority 3: Training on: counseling, and retention and graduation strategies. • Priority 4: Training to coordinate project activities with other available resources and activities, and training to design and operate a model TRIO project. • Priority 5: Training in the use of educational technology. Additionally, in the fall of2004, the TRIO Program Office established a quarterly newsletter for the purpose of sharing information and providing program guidance. The Winter 2004 issue of the TRIO newsletter discusses, among other things, the Training Program for TRIO Programs, reiterates the five training priorities, and announces the new recipients of the Training Program grants along with contact information. Finally, the TRIO Web site provides a wealth of information on the TRIO programs and is the primary mechanism for providing information about each of the TRIO programs including: program and funding requirements; regulations; law; performance reports; and other information. For example, the Talent Search Home Page lists the services that can be provided under the Talent Search Program and provides links to the program statute, regulations, etc. It is also important to note that senior TRIO staff routinely participate in policy seminars, state. regional, and national meetings hosted by grantees for the purpose of providing program information and policy guidance to grantees. Thank you for the opportunity to review and comment on this draft audit report. If you have questions about any of our comments, feel free to contact Larry Oxendine, Director, Federal TRIO Programs at (202) 502-7600 or Larry.Oxendine@ed.gov.
Talent Search Program at the U.S. Department of Education.
Published by the Department of Education, Office of Inspector General on 2005-02-16.
Below is a raw (and likely hideous) rendition of the original report. (PDF)