REPORT TO THE CONGRESS ‘4 Improvements Needed In The Administration Of Contracts For Evaluations And Studies Of Antipoverty Programs B.73o515 Office of Economic Opportunity BY THE COMPTROLLER GENERAL OF THE UNITED STATES 10954% 1 DEC. 28, a 97 2 COMFTROLLER GENERAL OF THE UNITED STATES WASWINGTON. D.C. 20548 B- 130515 To the President of the Senate and the .?’ Speaker of the House of Representatives This is our report on improvements needed in the ad- ministration of contracts for evaluations and studies of anti- I poverty programs of the Office of Economic Opportunity. 3’ -3 Our review was made pursuant to the Budget and Ac- counting Act, 1921 (31 U.S.C. 53), and the Accounting and Auditing Act of 1950 (31 U.S.C. 67). Copies of this report are being sent to the Director, Office of Management and Budget, and to the Director, Of- fice of Economic Opportunity. Comptroller General of the United States SOTH ANNIVERSARY 1921- 1971 Contents Pape DIGEST 1 CHAPTER 1 INTRODUCTION 5 Scope of review 5 REPORTSOF LIMITED OR QUESTIONABLE USEFULNESS 7 IMPROVED MANAGEMENTWOULDRESULT IN MORE USEFUL STUDIES 11 Need for clearer contract objectives and work descriptions 11 Need for improvements in monitoring con- tractor performance 14 Need for monitoring guidelines 14 Need for effective action on con- tract problems 15 Need for continuity in monitoring 18 4 NEED FOR A SYSTEMTO ENSUREASSESSMENTA.ND USE OF STUDY RESULTS 20 5 CONCLUSIONS, RECOMMENDATIONS,AND AGENCY COMMENTS 22 Conclusions 22 Recommendations to the Director, OEO 23 Agency comments 23 APPENDIX I Letter dated July 29, 1971, from the Deputy Director, Office of Economic Opportunity, to the General Accounting Office 25 II Summary information on selected contracts 28 III Principal officials of the Office of Eco- nomic Opportunity responsible for activi- ties discussed in this report 29 r ABBREVIATIONS _..^ GAO General Accounting Office OEO Office of Economic Opportunity VISTA Volunteers in Service to America * I CiiVPTROLLERGENERAL'S IMPROVEMENTS NEEDED IN THE ADMINISTRATION 1, KKPORTTO THE CONGRESS OF CONTRACTS FOR EVALUATIONS AND STUDIES OF ANTIPOVERTY PROGRAMS Office of Economic Opportunity B-130515 I WHYTHE REVIEW WASMADE 1 I The law directs the Office of Economic Opportunity (OEO) to provide for con- I tinuing evaluations of antipoverty programs, including their effectiveness I I in achieving stated goals. OEO may, for this purpose, contract for indepen- I dent evaluations of programs or projects. The law also authorizes OEO to contract for studies to test or assist in the development of new approaches I or methods to further the purpose of antipoverty programs. 1 In carrying out its functions, OEO has been a pioneer in advancing the state I of the art of evaluating social action programs, an important but complex and I difficult task. I I During fiscal years 1968 through 1970, OEO entered into 237 evaluation and 1 I study contracts amounting to about $30 million: The General Accounting Of- I fice (GAO) examined the management of these contracts because it wished to I determine whether useful results had been obtained, because there was a large I I amount of money involved, and because there was substantial congressional in- I terest in OEO's efforts. I ; FINDINGS AND CONCLUSIONS GAO reviewed 14 contracts amounting to $3.2 million, completed or scheduled to be completed during fiscal year 1970. Of these contracts, 10 did not fully achieve their purpose-- to provide OEO with objective and useful informa- tion to aid in the design, development, and assessment of its programs. OEO considered the contractor's reports under eight of the contracts as of no use or as falling short of intended objectives. Although OEO considered the reports under the six other contracts adequate and useful, GAO believes that the reports under two of these contracts were of questionable value be- cause the contractors' independence may have been compromised due to their involvement in the operations they were evaluating. (See p. 7.) GAO believes that several areas of OEO's contract administration need im- provement. Need for clear contract specifications Contract specifications did not always clearly and accurately describe the technical requirements of services to be procured, contrary to the Federal Procurement Regulations. The studies resulting from two contracts were not I Tear -- Sheet I I Prior to the completion of GAO's review, OEO issued instructions aimed at tightening the administration of its contracts. The new instructions should improve the management of OEO's evaluation and study contracts, but GAO believes that additional action is needed to effectively deal with the issues discussed in this report. (See p. 22.) RECOMMENDATIONS OR SUGGES!UONS OEO should: --Provide guidelines to assist OEO employees in selecting contractors that have the ability to make the required evaluations and studies in an objective and independent manner. --Issue detailed guidelines for use by OEO project managers in monitoring contractors' performances. I --Provide guidance and training to project managers to increase their abi'lity to effectively resolve problems encountered in contractor per- formance. --Ensure continuity in monitoring contractor performance by assigning i project managers to replace those who leave the agency or who are re- I assigned before the work under the contract is completed. I I I --Develop and implement a system to ensure (1) the formal assessment of ; the results of evaluations and studies and (2) the effective utilization of the results of contractors' studies to improve the designs and adminis- I tration of programs or to devefop new program approaches, when appro- I priate. (See p. 23.) ! I I AGENCYACTIONS AND UNRESOLVED ISSUES OEO acknowledged the problems discussed in GAO's report and agreed with the intent of its recommendations. OEO stated that a high-level task force had been convened which would deal with those problems identified in GAO's report that had not already been corrected. (See app. I.) MATTERSFOR CONSIDERATIONBY THE CONGRESS The Federal Government is making increased use of private contractors to evaluate and study its programs and related activities. This report illus- trates the need for exercising careful control over such contracts to en- sure that the results obtained are objective and usefuJ and are effectively utilized. I Tear Sheet 2 Y-3 INTRODUCTION The Economic Opportunity Act of 1964, as amended (42 U.S.C. 2701),commonly called the war on poverty act, authorizes the Office of Economic Opportunity to enter into contracts with public agencies and private organizations and persons to assist in carrying out the various require- ments of the act, Section 233 of title II of the act, which authorizes OEO to fund Community Action Programs and related programs, directs OEO to provide for continuing evaluations of the programs and authorizes OEO to contract for independent evaluations of (1) their effectiveness in achieving stated goals, (2) their impact on related antipoverty programs, and (3) their structures and mechanisms for the delivery of the required services. Section 232 authorizes OEO to con- tract for other studies to test or assist in the develop- ment of new approaches or methods that will aid in over- coming special problems or otherwise aid in furthering the purposes of title II of the act. During fiscal years 1968 through 1970, OEO awarded 237 evaluation and study contracts in the total amount of about $30 million. SCOPEOF REVIEW Our review was directed toward determining whether the policies, procedures, and practices followed by OEO in ad- ministering evaluation and study contracts were adequate to ensure that OEO was obtaining objective and useful informa- tion under these contracts and that OEO was making maximum utilization of final study reports. We selected for review 14 evaluation and study con- tracts totaling about $3.2 million, which had been recently completed or which were to be completed during fiscal year 1970. 5 Our selection of these contracts was based on a con- sideration of contracts with high dollar values, contracts that had been completed or that were to be completed during our review, and contracts representing a cross section of the various antipoverty programs administered by OEO but did not include 34 contracts for studies of Job Corps and Head Start projects, since these projects were no longer under OEOss management. We reviewed the records for the 14 contracts and dis- cussed them with present and former agency employees and with some of the contractors to determine whether the prod- ucts produced or being produced by the contractors were acceptable and useful to OEO. We also solicited comments from 10 of the 14 contractors. The comments received were considered by us in the preparation of this report. CHAPTER2 REPORTSOF LIMITED OR QUESTIONABLE USEF-ULNESS OEO has been a pioneer in advancing the state of the art of evaluating social action programs, an increasingly important but complex and difficult task. OEO's purpose in entering into evaluation and study contracts is to obtain useful and objective information to aid in the design, de- velopment, and management of its antipoverty programs. Our reviews however9 showed that, under several of these con- tracts, this purpose was not fully accomplished in that the reports submitted by the contractors were not useful or were of limited use for OEO program operations, We examined 14 contracts--eight awarded in fiscal year 1968 and six in fiscal year 1969--to determine the adequacy of OEO's administration of the contracts and its utiliza- tion of the study reports. Our interview of OEO officials and our review of available contract records and related correspondence for the 14 contracts showed that: --OEO considered the study reports obtained under six of the contracts, five of which were awarded in fis- cal year 1969, to be adequate and useful, Webelieve, however, that the reports obtained under two of the six contracts were of questionable value, because the contractors' independence may have been compro- mised due to their involvement in the operations they were evaluating. --QEO considered the reports obtained under the remain- ing eight contractsp seven of which were awarded in fiscal year 1968, as of no use or as falling short of meeting the intended objectives, Comments regarding these eight contracts follow. --A former director, Office of Special Field Programs, criticized as being unconvincing and lacking in ba- sic statistical credibility an evaluation costing $179,996 of the OEO migrant adult education program. He stated that "The work done to date lacks so much 7 credibility that it would never convince our grantees, our staff, the public and fee-X-* OEO." --A report on a study of planning development in Com- munity Action Programs made for the Office of Pro- gram Development at a cost of $151,166 was returned to the contractor for major revision at no additional cost to OEO. According to OEO officials the report failed to fulfill the requirements of the contract. Even after revisions were made, the Chief of the Evaluation Division commented that the report still contained "too much description and not enough evalu- ation." --A study,costing $285,7489of Volunteers in Service to America recruiting methods was considered of such poor quality that it served as a basis for rejecting the contractor's proposal for a later contract to do VISTA recruiting. The VISTA review panel reported: 'Yt'he final research report last year con- tained little more than a lengthy list of platitudes already known to Volunteers in Service to America's most junior recruit- ers." --A study, evaluation, and training contract in the amount of $383,276 for OEO's Office of Public Affairs was closed out at the request of the contractor be- fore it had submitted a final report. According to the Director of Procurement, problems which arose between the contractor and the OEO project manager during the administration of the contract could not be adequately resolved. The contract was terminated on January 5, 1971; the contract amount was reduced by $8,000, Reports or information pertaining to the other four contracts were similarly cited by OEO officials as lacking in expected quality; that is, as being too general or as being of limited usefulness because contract objectives had not been clearly defined. In commenting on a draft of this report, four contrac- tors took exception to OEO's evaluation of their work and 8 two did not take exception to OEO's evaluation of their work, Two contractors did not comment on our draft report. The reports on four of these eight contracts were dis- tributed to OEO program employees, Final reports for two other contracts were not distributed; another contract was terminated prior to its intended completion date, for which the contractor submitted only a status report summarizing its progress to the date of termination; and the remaining contract was completed without the contractor's issuing a final report. In addition to OEO's questioning the above eight con- tracts, we questioned the value of the work performed and the resultant reports under two contracts that OEO consid- ered adequate and useful, because the contractors were either participating in the operations of the OEO program which they were evaluating or providing technical aid to certain of the projects under the programs. Highlights of our observations on these two contracts follow. -0EO awarded in March 1969 a contract in the amount of $100,350 for the evaluation of VISTA programs to a contractor which at the time had contracts of about $5 million under which he was carrying out certain operations for VISTA. In commenting on our draft re- port, the contractor stated that the consultants who had performed under the contract were independent agents and that there was no question as to the ob- jectiveness and usefulness of the study reports. Although the effect that these other contracts had upon the judgment of the consultants employed by the contractor could not be determined, we believe that, to ensure objective and useful study reports, OEO should select contractors for evaluations that are independent of the programs to be evaluated. --Under a contract awarded in October 1967 in the amount of $440,734 for OEO's Office of Program Development, a contractor was required to provide technical assis- tance to rural research and demonstration projects and at the same time to monitor and evaluate the projects. Both the Assistant General Counsel and the Chief of the Rural Branch agreed that the contractor should not have been allowed to determine the 9 technical assistance needed and to be provided to the proj- ects and also to monitor and evaluate the technical assis- tance it provided. Tn our opinion, these are requirements which, when not met independently, represent a conflict of interests. In April 1970 OEO issued instructions which included guidelines for preventing conflict-of-interest situations in contracting. The guidelines deal principally with pre- cluding contract awards to organizations that employ former OEO employees. They do not, however ) provide guidance to assist contract planners in selecting contractors that can do the required contract work independently and objectively and that are not otherwise involved in OEO activities which could impair this ability. In commenting on a draft of this report, OEO stated its belief that, to judge the value of performing evaluation studies, the cost of the studies should be compared with total program expenditures. In OEO's opinion, it is con- ceivable that the beneficial impact of one successful evalu- ation could more than justify the overall expenditure of its evaluation efforts. We recognize that some evaluations may result in greater program benefits that others, but we believe that successful evaluations cannot justify defects in others which may have been caused by weaknesses in the administration of the eval- uation contracts. Additional information on the 14 contracts considered in our review, including a summary statement on the useful- ness of the studies, is shown in appendix II of this report. 10 CHAPTER3 IMPROVEDMANAGEMENTWOULDRESULT IN MORE USEFUL STUDIES Frocedural problems weakened OEO's control over the work done under the contracts we reviewed and lessened the likelihood of receiving useful study results. --Contract objectives and statements of work should have been more clearly defined. Vagueness in objec- tives caused misunderstandings between project man- agers and contractors. --The methods used to monitor studies were left largely to the preferences of individual project managers. Contract problems were not identified and resolved in an effective mmer. NEED FOR CLEARER CONTRACTOBJECTIVES AND WORKDESCRIPTIONS The Federal Procurement Regulations, under which these contracts were negotiated, require that all parties to a contract obtain a good understanding of its objectives and scope. Implicit in this requirement is the need for agency employees to adequately define contract objectives and to draft contract specifications which must be met to achieve the objectives. According to section l-1.307-2 of the Fed- eral Procurement Regulations, contract specifications should clearly and accurately describe the technical requirements of the products or services to be procured. For two contracts the resulting studies were not satis- factory because of problems in initially determining con- tract objectives and formulating work descriptions. In the first case OEO awarded a contract in the amount of $285,748 for a study of VISTA recruiting methods. The objectives of the contract were not clearly established either prior to or after the contract award. Our inquiries of OEO employees associated with the contract revealed that there was disagreement among VISTA officials as to whether the contract was for the actual recruitment of VISTAvolun- teers or for research or evaluation of VISTA recruitment techniques. An associate director of VISTA, who had assumed respon- sibility for the contract after work began, told us that the original purpose of contract was to help recruit more vol- unteers but that, because the then Director of VISTA did not favor a recruitment contract, it had been awarded as a re- cruiting-research effort. In a final evaluation of the work done under the con- tract, VISTA officials concluded that (1) as a recruiting effort, it had fallen short of VISTA's own recruiting re- sults during that same period and (2) as a research effort, the contractorDs final report was less than satisfactory. In the other case, which concerned a contract in the amount of $329,355 for a study of the Community Action Pro- gram in a selected county of Texas, the work description was not sufficiently detailed to reasonably ensure that the contract objectives would be accomplished. This contract was to be phase II of a two-phased study of the Community Action Program in Austin and Travis County, Texas. Both phases were to produce results which could be combined with the results of similar studies being conducted in other geo- graphical areas in an attempt to develop an overall strategy for evaluating OEO's Community Action Program. According to a former chief of the Evaluation Division, who had been responsible for managing this contract, OEO officials decided upon an additional major objective for phase II of the study--that the contractor formulate sugges- tions for improving the Texas program. The work description for phase II, however, did not require final recommendations by the contractor, contrary to what had been contemplated. As a result, after the con- tractor's final report was received, the Chief of the Bvalua- tion Division requested a letter from the contractor setting forth recommendations based upon the 3-year, two-phased study which cost OEO about $670,000. In its reply, the con- tractor pointed out that: 12 "In atterilpting to arrive at policy recommendations after our 3-year study of the CAP [Community Ac- tion Program] effort, the inescapable conclusion is that, as social scientists, we are better able to define problems than we are at recommending program changes." Nevertheless the contractor provided a list of suggestions and observations which it called off-the-cuff comments. According to OEO officials, these comments were not fur- nished to persons responsible for managing the Community Action Program in Texas for their use. In April 1970, recognizing the need for clear contract objectives and specific work descriptions, OEO issued to its headquarters staff instructions requiring that contract work descriptions be in such detail that OEO could hold the contractors accountable for exactly that for which OEO had contracted. 13 NEED FOR II%!?ROVEMENTS IN NONITORING CONTRACTORPERFO~?' Improved monitoring of contractor perfo-rmance was needed to ensure that contractors' work was progressing satisfactorily. The responsibility for contract monitoring is assigned to an OEO official designated by the contracting officer to act as the project manager. The contracts usually described the project manager and his authority as follows. "A representative of the Government, designated by the Contracting Officer ***. He has full authority to represent the Contracting Officer in connection with the operations under this con- tract, except that, he is not authorized to issue orders which change the work to be performed here- under, the compensation, or the period of perfor- mance." The effectiveness of a project manager in overseeing and guiding the contract work can contribute significantly to the success or failure of the contractor's efforts. OEO's April 1970 instructions established the policy for the selection of project managers and broadly described their role and authority. The guidelines, however, did not adequately provide for (1) monitoring contractor perfor- mance, (2) dealing effectively with problems encountered in contractor performance, and (3) maintaining continuity in monitoring contracts when project managers responsible for monitoring the contracts left the agency or were reassigned. Need for monitoring guidelines Effective monitoring requires that project managers keep fully informed of contractors' progress and any diffi- culties encountered in meeting contract objectives. Good monitoring generally includes --visiting contractors' sites, --obtaining and analyzing reports from the contractor regarding the progress being made, and 14 --maintaining formal records of agreements reached with the contractor. In the absence of specific guidelines, contract moni- toring was largely dependent upon the preferences, abili- ties, and work loads of the individual OEO project managers. As a result, progress reports in some cases were not re- quired or did not contain meaningful data; site visits gen- erally were not made; and basic data on contractor perfor- mance, such as the project manager's assessment of the con- tractor's work progress and achievement of interim goals in many instances was not available in the contract files. For example, OEQ awarded a contract which was com- pleted on June 30, 1969, at a cost of $179,996, for an eval- uation of the adult education and training program for mi- grants. The contract required that progress reports be submitted every 30 days to OEO. According to the project manager, the reports were not adequate for use in monitor- ing contractor performance because they were too vague re- garding work progress and problems encountered by the con- tractor. Although the project manager considered the contrac- tor's progress reports inadequate, he did not visit the worksites while the contract was being performed. Instead, he relied on the contractor's initial working plan, the progress reports, and some telephone communication with the contractor. These means of monitoring contractor perfor- mance proved inadequate as the final study report submitted by the contractor was considered useless by OEO officials. Need for effective action on contract problems In two of the contracts included in our review, prob- lems which adversely affected the progress and outcomes of the studies were not effectively dealt with by the project managers. In the case of the contract to evaluate education and training programs for migrants, the project manager in- formed us that in February 1969 he first began to realize that the contractor was not developing the statistical data required by the contract. The project manager considered this data necessary to draw valid study conclusions and to make the final report responsive to agency needs. Payments continued to be made to the contractor, however, during the balance of the H-month contract period to June 30, 1969, by which time $167,493 of the contract amount of $179,996 had been paid to the contractor. In July 1969, 5 months after the project manager realized the problems, the con- tractor was for the first time formally notified by an OEO official that the work was inadequate and was not useful to OEQ. In our opinion, had the project manager been adequately monitoring this contract, he might have (1) recognized at an earlier date that the contractor's performance was un- satisfactory and (2) obtained the assistance of other CEO offices, such as the Office of General Counsel or the Fro- curement Division, to determine whether payments under the contract should have been discontinued until the problems were corrected. AS a result of our inquiries, the contract was brought to the attention of the General Counsel of OEO and the ap- propriate procurement officials. As of November 1971 CEO was considering what, if any,action would be taken to re- solve the contract problems. In another case, which involved a study, evaluation, and training contract undertaken for the OEO Office of Public Affairs, the project manager did not ask for assis- tance from the Procurement Division or the General Counsel even though problems were encountered which warranted such attention. In this case the contractor failed to provide a full-time project director and did not carry out normal ad- ministrative functions to support the project. The project manager was not successful in resolving these problems, and the contractor eventually contended that it was unable to complete the work as contemplated. The basis for the contractor's complaint was detailed in a letter to OEO's Procurement Division. "Because of repeated changes of instruction by the OEO Project Manager, and repeated changes in the conception of the project it has proven im- possible for us to fulfill all of the physical requirements of the contract. This is a re- quest to be relieved of certain requirements, with appropriate adjustments, so that the contract may be terminated." OEO agreed to the contractor's request, and on Janu- ary 5, 1971, the contract was terminated before the final report was prepared. Ihecontract amount was reduced by $8,000. OEOps system of making progress payments to contrac- tors did not, at the time of our fieldwork, require the con- current approval of the project manager. Periodic billings were sent by the contractor to the Defense Contract Audit Agency1 for a routine fiscal check and from there were for- warded to OEO's Finance Division for payment. Only the final payment to the contractor required the project man- ager's specific approval which 'was to be based on the satis- factory completion ot the contract work. The Procurement Director of OEO informed us that a double vouchering system would more actively involve project managers in the approval of progress payments. Under such a system a contractor would send duplicate vouchers to the Defense Contract Audit Agency and the OEO project manager for their respective approvals. The Defense Contract Audit Agency would continue to be responsible for determining the validity of the contractor's costs, and the project manager would certify to the quality of the contractor's work. We believe that a system requiring the project manager's ap- proval of progress payments would strengthen his role in monitoring contractor performance. In addition, project managers needed to be better ac- quainted with OEO's administrative procedures for dealing with problems involving contractor performance. Project managers generally were chosen from program employees having technical backgrounds in the social sciences and having 1 The Defense Contract Audit Agency acts as the cognizant audit agency for most OEO contracts. 17 little orientation in procurement, financial administration, and legal matters. Therefore OEO, through guidelines and training, should acquaint project managers with the special- ized assistance available to them in dealing with problems of contract administration. Need for continuity in monitoring For the management of evaluation and study projects to be effective, OEO must ensure that project managers are qualified to monitor contractor performance of a project and that continuity of responsibility for the project is maintained when project managers leave the agency or o%her- wise relinquish responsibility for the project before it 5s completed. In the following two cases, OEO did not ap- point project managers to replace those who left. Conse- quently OEO was not in a position to monitor the contract work and resolve problems precluding the fulfillment of contract objectives. The first case involved a contract for $100,000 for a survey of mental health care programs to which a project manager was no% assigned during the greater part of i%s duration, The contract, which was awarded on June 24, 1968, originally designated a project manager, but he left OEO soon after the award of the contract, A replacement was not made by the OEO contracting officer. The contractor experienced difficulties in doing the contract work, espe- cially in organizing and controlling the survey, As a re- sult, the contract was in effect for about 14 months, dur- ing which time virtually no work was done. On June 26, 1969, and again, on July 14, 1969, the contractor requested that the project be terminated. The contractor9s July 14, 1969, request included the following statement: I'** at this point in time %he contract project is not as valid as when originally conceived. Therefore, taking everything into consideration, it is the opinion of the [contractor] that it is not in the best interest of the Government to continue the contract and recommends that it be cancelled." In September 1969 OEO terminated the contract. The con- tractor claimed costs of $25,000, which, as of September 1971, were subject to a final settlement. We believe that, if an OEO project manager had been assigned, he would have been in a position to recognize the contractor's problems and seek earlier resolutions of the problems. The unused portion of the $100,000 earmarked for this project might have been released earlier and might have been put to use on a more worthwhile antipoverty ef- fort. In the case of a study of the summer Youth program which cost $100,408, OEO did not assign a new project man- ager after the designated project manager left the agency. The need for continuity in monitoring this study was partic- ularly critical because (1) the project manager, before leaving OEO, directed the contractor to revise its report, at an additional cost of $2,200 to satisfycontract require- ments and (2) during the period of the contract, the summer Youth program was transferred to a newly created OEO pro- gram office. As a result of not appointing a new project manager, OEO lost control over the study and considered the final report to be of lower quality than expected. The re- port, however, was used as a basis for developing new pro- gram guidelines. 19 CHAPTER4 -..._-_ NEED FOR A SYSTEM ----.d.-.-*_q TO ENSUREASSESSMENTAND USE OF STUDY RESULTS -- Although the effectiveness and usefulness of evalua- tions and studies undertaken by OEO contractors depend on the extent to which the results of such studies can be, and actually are, used, we noted that OEO had no system for ensuring that the validity of study results was formally assessed and that valid contractor recommendations were used for the intended management purposes. For 12 of the 14 contracts, final reports were submit- ted by the contractors; however, most of the reports were not formally assessed to determine their usefulness, Al- though informal appraisals of the general acceptability of the reports were made by individual project managers, OEO project files did not contain documentation of systematic assessments of contractors' conclusions and recommendations and the reasons for taking, or not taking, action on them. Further CEO did not have a system to ensure that appropri- ate action would be taken on all study findings, conclusions, and recommendations consideredworthwhileor to review im- plementation of any actions taken as a result of the studies, OEO has recognized that there is a need for establish- ing a system for assessing and utilizing the results of study reports. In June 1969 OEO awarded a contract for $26,293 for the development of a system for disseminating the data developed under evaluations and studies within both OEO headquarters and l&al Community Action Agencies. The request for proposals for the contract pointed out the following deficiency within OEO. llCAP [Community Action Program] program evalua- tions and evaluation systems have been developed on a piece meal basis. These systems are provid- ing piece meal information to policy makers and program operators. Little is known about what happens to evaluation information and there has been no assessment of the value of evaluation in- formation that has been produced or the methodol- ogy that has produced the infomation.t' 20 The contractor confirmed this deficiency by stating in his final report that: "Although we certainly don't have the data to sup- port any generalizations, our interview did rein- force the impression that utilization of evalua- tion results in general was mostly an accidental consequence of an interest in a program somewhere in OEO and information about that program appear- ing somewhere else within the Agency." The contractor recommended an extension of its con- tract for t'he purpose of developing a prototype handbook for use by OEO officials involved in initiating, monitoring, and assessing evaluative research efforts; a prototype pam- phlet for use by potential contractors and grantees in pre- paring proposals and interim and final reports; and a pro- totype brochure for non-research-oriented employees explain- ing in simple terms the basics of research in antipoverty programs. In April 1970 OEO awarded the contractor a follow-on contract in the amount of $30,967 to develop for use by OEO officials standards and guidelines for preparing, monitoring, and assessing evaluation projects. In March 1971 OEO re- ceived the contractor's final report and as of September 1971 was utilizing the report in developing new procedures and regulations designed to increase the effectiveness and utilization of OEO evaluation projects. The final assessment of study results and, where ap- propriate, the issuance of directives for implementing con- tractor recommendational level above the project managers. OEO should consider using panels of top-level officials, such as individual office and program directors and assis- tant and associate directors, who would have responsibility for formally assessing and, where appropriate, initiating action on contractors' recommendations. The panel approach might utilize one of the several boards which OEO has estab- lished to review proposed procurement actions and to evalu- ate contractor proposals. 21 CONCLUSIONS, RECOMMENDATIONS,AND AGENCYCOMMENTS CONCLUSIONS OEO needed to make substantial improvement in the planning and monitoring of evaluations and studies to ob- tain useful information commensurate with the high cost of these evaluations and studies. Prior to the completion of our fieldwork, OEO developed instructions aimed at tightening the overall administration of its contracts and grants, Some of the instructions, is- sued in April 1970, deal with contract planning, the prep- aration of work descriptions, the use of OEO review panels to consider contract proposals, and the preparation and maintenance of contractor performance records. Other in- structions deal with avoiding conflicts of interest and strengthening the functions of OEO project managers. The new instructions, if properly implemented, should improve the management of OEO contracts for evaluations and other studies; however, the instructions do not provide needed guidelines for (1) ensuring the independence and objectivity of contractors, (2) adequately monitoring contractor per- formance, (3) dealing effectively with problems encountered in contractor performance, or (4) maintaining continuity in monitoring contracts. Special emphasis is needed in these areas of OEO contract management to correct the deficiencies identified during our review and to prevent them from re- curring. Also, although evaluations and studies were to give OEO information it needed for improving antipoverty efforts, OEO had no system to (1) assess the usefulness of contractors' conclusions and recommendations, (2) initiate action to bring about needed changes, or (3) follow up on actions taken as a result of studies to determine whether the ac- tions had been properly implemented and were effective. 22 RECOMMENDATIONS TO THE DIRECTOR, OEO We recommend that OEO: 1. Provide guidelines to assist OEO employees in select- ing contractors that have the ability to make the required evaluations and studies in an objective and independent manner. 2. Issue detailed guidelines for use by OEO project managers in monitoring contractors' performances. The guidelines should require project managers to visit contractors' sites to observe performance, obtain progress reports, and fully document agree- ments or other understandings reached between the project managers and the contractors. 3. Provide guidance and training to project managers to increase their ability to effectively resolve problems encountered in contractor performance. Project managers should be instructed to seek the assistance of OEO contracting, legal, and other specialists as needed to safeguard OEO's contractual rights and interests. 4. Ensure continuity in monitoring contractor performance by assigning project managers to replace those who leave the agency or who are reassigned before work under the contract is completed. 5. Develop and implement a system to ensure (a> the formal assessment of the results of evaluations and study contracts-- possibly by panels of top-level OEO officials-- and (b) the effective utilization of the results of contractors' studies to improve the de- signs and administration of programs or to develop new program approaches, when appropriate, AGENCY COMMENTS OEO, in commenting on a draft of this report in a letter dated July 29, 1971 (see app. I), stated that it recognized the problems discussed in the report and concurred in the intent of our recommendations. 23 OEO stated also that the following actions had been or were being taken to improve contract management. 1. A high-level task force on program management, con- tracting, and grant issuance was convened to deal with those problems identified in our report and in other OEO studies that had not already been corrected. 2. More attention was being given to source selection criteria and methodoloa to ensure that, after price and all other factors were considered, the best possible contractor was chosen. 3. Attempts were made to phase out many demonstration and other small projects which should reduce the number of evaluation studies associated with those projects and which should thereby reduce the work- load of the limited staff of the Evaluation Division. 4. Additional employee slots were acquired to increase the size of the evaluation staff, and additional positions for this staff were requested. 24 APPENDIX I EXECUTIVE OFFICE OF THE PRESIDEW OFFICE OF ECONORIIC WASWINGYON. D.C. 20506 July 29, 1971 JIJL 29 1971 Mr. Henry Eschwege Associ'ate Director Civil Division U. S. General Accounting Office Washington, D. C. 20548 Dear Mr. Eschwege: Thank you for the opportunity to comment on the draft report, "Stronger Management Needed to Increase the Effectiveness of Evaluations of Studies of Anti-Poverty Programs." The report is interesting and useful to us and other agencies engaged in performing evaluations of social action programs. The problems discussed are real and we concur with the intent of the recommendations. As noted in the report, we had already taken action to eliminate some of the problems. I have recently convened a high level task force on Program Management, Contracting and Grant Issuance whose charter is to deal with remaining problems identified in your report and in other studies we have made. Your report should have some additions to avoid misinterpretation and to increase the usefulness of the recommendations to those others who will use it to deal with problems in management of evaluation studies. Your review of our evaluation studies was directed at,improving them: We are concerned, however, that the results are subject to misinterpretation. To some extent this problem cannot be avoided. It could be easier for responsible people to understand the context for evaluation and the difficulty of drawing conclusions about its value from studies such as yours, if your report pointed out such things as the following: a) From a contract management point of view, the number of inadequately managed contracts is important. In judging the value of performing evaluation studies, however, the cost of the studies has to be compared with total program expenditures. It is entirely conceivable that the beneficial impact of one successful evaluation could more than justify the overall expenditure of our evaluation efforts. b) Since many readers areilikely to generalize from your results in spite of whatever cautions you might put in, it is desirable to balance the generally negative picture that the numbers in their current form represent by pointing out that four of the six contracts let after June 1968 were considered to be acceptable by GAO, and one more was considered useful by the Office of Economic Opportunity. None let before July 1968 were considered acceptable by GAO. 25 APPENDIX I c> One of the reasons that good evaluations are difficult to per- form is that it is almost impossible to find a “normal” period in the operation of a program. You were forced to evaluate the program as it existed at the time that you conducted your evaluation. Nevertheless, it should be recognized that 1969 represented a year of very rapid change both in philosophy and in personnel, even by our standards, This change was probably responsible for at least some of the discontinuity in management. d) We understand that Job Corps and Head Start studies were not reviewed because the programs are no longer managed by the Office of Economic Opportunity. Spinning off these programs reflected a judgment on program and management maturity. As one indicator of this, both programs had quite well developed and systematized evaluation programs. Since the 235 contracts and funding cited appear to include contracts let by Job Corps and Head Start, the reader needs to be cautioned about the effect on any generalization from your results to an overall measure of evaluation contract success. Your report recommends the institution of a number of improved management procedures. There is no question that such a system helps to improve the quality of evaluation studies. It is, however, of secondary importance, as compared to the existence of an appropriately qualified staff whose workload is limited so that there is adequate time for them to do a good job. In particular : 4 The comparatively rudimentary state of development of evaluation measures makes it very difficult to design a study which will yield useful results in areas of immediate interest to decision makers. The people who do this work should be trained and experienced analysts with knowledge of social action programs and research procedures relating to the social sciences. Because such people are very scarce, one can, instead, use teams of at least one experienced analyst with statistical or operations research training, one researcher with appropriate social science skills and additional consultants where necessary. b) Many of the studies are sufficiently large in scope to require considerable management competence on the part of the project manager. This competence is most likely to be developed through prior management experience, For a variety of reasons, in- cluding the need for training; for continuity during vacations, illness or departure of the project manager; and because people appear to work more effectively in teams, at least two pepple should be assigned to every study. At least one of them should be an experienced manager. The number of studies which can be effectively assigned to an individual depends on their scope, but should not exceed two or three. If the study is complex and important enough, it should be the only assignment for the team. 26 Effective Cost Responsible Ilseful,;e>s of c,,d product ( 1 i) _?Ys dates (-a) OEO office Purpoze and other <onmeuts A Flxed price 10-18-67 S 440.734 Proeram Devel- Evaloatio" alld fea- The reports were considered useful by bol?e OEO officials. opment sibility studies of In our opinion, however, their objectivity was question- 5x70 rural research and able because the contractor was required Lo monitor and demonstration proj- technically assist the same projects. According to the ects. OEO General Counsel, these requirements conflicted. B cost plus 2- 8-60 285,748 Volunteers Evaluation of VISTA The final research report was considered useless by OEO fixed fee in Service recruiting methods. officials and was described as informaria already known 9-::-60 to America by VISTA junior recruiters. c Cost plus 4-15-60 329,355 Planning, Study and evaluation The COI~S~IISUS of OEC! officials contacted was that the re- fixed fee Research and of Community Action port was of limited usefulness. The report, along with 12-:00-69 Evaluation Program in Austin other Community Action Program evaluation studies, was and Travis County. submitted to the University of Wisconsin for further re- search. D Cost plus 6-10-68 100,408 Program Devel- Evaluation of summer The last of two re"orts was revised twice at additional fixed fee opment Youth programs. cost to OEC. The 'final version was considered of less 5-::-69 than expected quality by OEO officials but reportedly was useful in developing program guidelines. E Cost plus 6-24-68 100,000 Health Affairs Survey of psychiat- The contract was terminated, and a status report was sub- fixed fee tcl ric mental health mitted by the contractor which was, according to OEO of- 4-u-70 care programs. ficials, of limited usefulness. Final contract settle- ment was still pending as of September 1971. F Cost plus 6-29-68 179.996 Program l?evel- Evaluation of adult The report was considered useless by OEO officials be- fixed fee opment education and train- cause the contractor did not satisfactorily develop the 6x69 ing programs for mi- required statistical data. The reported conclusions and grants. recommendations, therefore, were considered untenable by OEO. G Fixed price 6-28-68 362,069 Program Devel- Evaluation and re- Five of the six final reports were characterized by OEO opment search. officials as marginal in quality and as lacking in depth 7-::-69 and perception. The usefulness of the information was reported by OEO as being minimal, and the contractor was cited as being ""able to provide the quality of work ex- pected. H Fixed price 6-29-68 383,276 Public Affairs Study, evaluation, There was "a final report--due to OEO project management and training program difficulties, OEO terminated the contract at the request 1x70 and production of of the contractor before all contract requirements were 16 mm films. met. The contract amount was reduced by $8,000. I Cost plus lo- l-68 471,774 Planning, Evaluation of Cornmu- The final report was considered good by OEO and was used fixed fee Research and nity Action Pro- as a basis. along with reports of similar contracted 7&69 Evaluation grams. studies, to award a follow-o" contract to the same con- tractor for analysis of a.11 the Community Action Program data that had been collected. J cost plus 4- 9-69 104,452 Operations Feasibility of as- The final report was considered satisfactory by OEO offi- fixed fee to sessing the impact cials and was used for examination and correction of man- 8-15-69 of training and agement problems in the OEO Training and Technical Assis- technical assistance tance Division. A follow-on contract was proposed to de- in Community Action sign and field test a system for measuring the impact of Programs. training in the OEO regions. K cost plus 3- l-69 100,350 Volunteers Evaluation of VISTA The reports were considered useful by OEO; however, in fixed fee to in Service programs. our opinion, their objectivity was questionable in that Z-28-70 to America the contractor hired for the evaluation had other large contract interests in the VISTA program. L cost plus 6-24-69 127,812 Planning, Report on studies of The final reports were considered satisfactory by OEO and fixed fee Research and Upward Bound pro- were furnished to the Deparunent of Health, Education, 2-z-70 Evaluation gram. and Welfare which currently is responsible for the Upward Bound program. M Fixed price 6-27-69 26,293 Program Devel- Assessment of Commu- The contract was awarded, in pert, to help OEO develop an to opment nity Action Program Internal utilization system for its evaluation and study 11-15-69 evaluations and de- information. OEO considered the contract useful in that sign for dissemina- the contractor pointed at system deficiencies. The con- tion of information. tractor recommended contract extension, and a follow-on contract was awarded. N Cost p1xs 6-27-69 151,163 Progran Devel- Evaluation of the Tne final report draft was revised at no cost to OEO. flxed price 0pSEIlt Community Action The revised report was critiqued by OEO officials as con- 2-::-70 $3 163 433 ==!zc=zL Program planning de- taining too much description and not enough evaluation velopment program. and, hence, as of limited usefulness. The co"tracfar'* overall performance was stated as less than satisfactory. ?n most cases these figures represent estimates as shown I" the contracts, many of which were still pending final audit and settlement at the completion of our fieldwork. APPENDIX III PRINCIPAL OFFICIALS OF THE OFFICE OF ECONOMIC OPPORTUNITY RESPONSIBLE FOR ACTIVITIES DISCUSSED IN THIS REPORT Tenure of office ---- From To DIRECTOR: Phillip V. Sanchez Sept. 1971 Present Frank C. Carlucci Dee, 1970 Sept. 1971 Donald Rumsfeld %Y 1969 Dec. 1970 Bertrand M. Harding (acting) Mar, 1968 &Y 1969 R. Sargent Shriver Oct. 1964 Mar, 1968 DEPUTY DIRECTOR: Wesley L. Hj ornevik Oct. 1969 Present Robert Perrin (acting) Mar. 1968 Oct. 1969 Bertrand M, Harding June 1966 Mar. 1968 ASSOCIATE DIRECTOR FOR ADMINIS- TRATION (note a>: Ernest Russell (acting> Apr. 1971 Present Robert C. Cassidy Sept, 1967 Apr. 1971 ASSISTANT DIRECT0R FOR PLANNING, RESEARCHAND EVALUATION (noteb): John 0. Wilson Oct. 1969 Present Richard Ottman (acting) Jan, 1969 Sept. 1969 Robert A. Levine Nov. 1966 Jan. 1969 ASSISTANT/ASSOCIATE DIRECTOR FOR PUBLIC AFFAIRS: Louis Churchville (Associate) June 1970 Present Richard D, Peacock (acting) Feb. 1970 June 1970 Arch M&inlay Aug. 1969 Jan. 1970 Kenneth Sparks (acting) June 1968 Aug. 1969 Herbert J. Kramer Jan. 1966 June 1968 29 APPENDIX III Tenure of office -.--1-1_ From --- -To ASSISTANT DIRECTOR FOR PROGRAM DEVELOPMENT: Carol Khosrovi July 1971 Present Alfred Taylor (acting) June 1971 July 1971 Joseph P. Maldonado Aug. 1970 June 1971 Marvin J. Feldman Jan. 1970 Aug. 1970 Robert Perrin (acting) Sept. 1969 Jan, 1970 Theodore M. Berry (Assistant Director for Community Ac- tion Program) (note c) Apr. 1965 Sept. 1969 ASSISTANT DIRECTOR FOR OPERATIONS: H. Rodger Betts (acting) Sept. 1971 Present Phillip V. Sanchez Feb. 1971 Sept. 1971 Donald I. Wortman (acting) Dec. 1970 Feb. 1971 Frank C. Carluoci Dec. 1969 Dee, 1970 William Bozman (acting> Oct. 1969 Dec. 1969 Theodore M, Berry (Assistant Director for Community Ac- tion Program (note c> Apr. 1965 Sept. 1969 ASSISTANT/ASSOCIATE DIRECTOR FOR HEALTH AFFAIRS: Carl A. Smith (acting) MY 1971 Present Thomas E, Bryant (Associate) Sept. 1969 Apr. 1971 Thomas E. Bryant (acting> Jan. 1969 Sept. 1969 Joseph T. English Mar. 1968 Jan. 1969 ASSISTANT DIRECTOR FOR VOW&IT IN SERVICE TO I@4 (note d): Carol Khosrovi Sept. 1970 July 1971 Chester R. Lane (a.eting) Jm. 1970 Sept. 1970 Padraic Kennedy (acting> July 31968 Jan. 1970 Bill Crook Mar. 1967 June 1968 30 APPENDIX III aPrior to June 1968 this office was called the Office of Management. b Prior to 0EO"s September 1969 reorganization, this office was called the Office of Research, Plans, Programs and Evaluation. 'In September 1969 this position was terminated as an or- ganizational entity and-responsibility for the programs was shifted to two newly created offices, the Office of Program Development and the Office of Operations, dp ursuant to Reorganization Plan No. 1 of 1971, on July 1, 1971, the VISTA program was transferred from OEO to ACTION, a new Federal agency. U.S GAO, Wash., D.C. 31 ,\ ,-, , - _- - .. =,.: d,,. ., ,, --r .. - -, ! ‘L Copies of this report are available from the U. S. General Accounting Office, Room 6417, 441 G Street, N W., Washington, D.C., 20548. Copies are provided without charge to Mem- bers of Congress, congressiona I committee staff members, Government officia Is, members of the press, college libraries, faculty mem- bers and students. The price to the general public is $1 .OO a copy. Orders should be ac- companied by cash or check.
Improvements Needed in the Administration of Contracts for Evaluations and Studies of Antipoverty Programs
Published by the Government Accountability Office on 1971-12-28.
Below is a raw (and likely hideous) rendition of the original report. (PDF)