Criteria Needed For Measuring Technical Assistance Results And Contractor Performance Et-160789 Agency for International Development Department of State UNITED STATES GENERAL ACCOUNTlNG OFFICE UNITED STATES GENERAL ACCOUNTING OFFICE WASHINGTON. D.C. 20548 iNTERNATIONAL OlVlSION B-l 60789 Dear Dr. Hannah: “-i-; This report highlights some of the accomplishments and problems observed during our review of the Agency’s use of contractors on tech- nical assistance projects in Kenya, Tanzania, and Uganda. These obser- vations are summarized in the digest included in this report. A draft of this report was submitted to your Agency for comment. The response, included as an appendix, was considered in the prepara- tion of this report. In the draft we proposed that AID: 1. Take appropriate action to require in actual practice that the project planning and programming documentation and the tech- nical service contracts contain more specific delineations of scope of work, goals, and objectives and quantitative indicators by which progress may be evaluated. 2. Institute a procedure for measuring on a continuing basis (a) the seriousness and causes of contract staffing problems, including the identification of contractors who fail to pro- vide staff on a timely basis and (b) the full significance of the nonavailability for training of host country nationals. We have been advised that, after our fieldwork, the Agency has revised its technical assistance evaluation system and that “AID has expended a sizeable amount of effort in reorganizing, planning, and es- tablishing documentation systems in order to correct deficiencies.” We do not know what effect these changes have had in actual practice. 50 TH ANNIVERSARY 1921-1971 ’ B-160739 We recommend that you obtain an independent evaluation of the current practices and procedures and determine the extent to which the types of weaknesses pointed up in this report have been overcome in actual practice. Your attention is invited to section 236 of the Legislative Reorga- nization Act of 1970 which requires that you submit written statements of the action taken with respect to the recommendation. The statements are to be sent to the House and Senate Committees on Government Op- erations not later than 60 days after the date of this report and to the House and Senate Committees on Appropriations in connection with the first request for appropriations submitted by your Agency more than 60 days after the date of this report. We would appreciate receiving /;. -- copies of all statements submitted. 1, Copies of this report are being sent today to the above committees; the Subcommittee on Foreign Operations, Senate Committee on Appropri- ations; the Foreign Operations and Government Information Subcommit- tee, House Committee on Government Operations; the Secretary of State; and the Director, Office of Management and Budget. We wish to acknowledge the cooperation extended to our represen- tatives during the review. Sincerely yours, Oye V. Stovall Director The Honorable John A. Hannah Administrator, Agency for International Development 2 ; . GENEML ACCOUNTING OFFICE CRITERIA NEEDED FOR MEASURING TECHNICAL I REFORT-TO THE ADMINISTRATOR ASSISTANCE RESULTS AND CONTRACTOR I I AGENCY FOR INTERNATIONAL PERFORMANCE I DE.VELOFMENT Agency for International Development I Department of State B-160789 I I I DIGEST ------ I I I WHY THE REVIEW WAS MADE I I I Contracts are an important instrument of the Agency for International I Development (AID) in providing technical assistance. As of December 31, I I 1970, AID had 1,191 contracts outstanding totaling about $679 million I for technical assistance in 58 countries. I I I Because of the dollar amounts expended and the extensive use of this I means of providing technical assistance, the General Accounting Office I (GAO) examined AID’s use of technical service contracts for 10 agricul- I tural projects in Kenya, Tanzania, and Uganda. I ---*--. -- - In carrying out technical assistance projects, AID contracts with educa- tional institutions and qualified firms, associations, and individuals. U.S. universities have been among AID’s primary sources. FINDINGS AND CONCLUSIONS Neither AID’s project planning and programming documents nor the contracts themselves provided specific targets, goals, or objectives. AID therefore could not evaluate progress and contractor performance objectively. Notwithstanding the absence of objective measurement criteria, some positive and significant results have been achieved through AID’s technical service contracting. For example: --AID was instrumental in introducing vocational agricultural education in Kenya's secondary schools. (See p. 16.) I --AID assisted the Government of Uganda to build and expand a strong and I I independent cooperative movement, particularly in the dairy industry I and in livestock ranching. (See p. 16.) I I Various problems, however, have hindered the effectiveness of AID’s efforts. I I There were delays in recruiting and placing contract employees in-country I and some employees appeared to lack desired qualifications. (See p. 19.) I I I Other problems limiting the effectiveness of technical assistance were re- I I lated to host country actions. For example, preparing host country na- I tionals as replacements for contract employees is an important objective I of the projects. This objective was not fully achieved because host I I I I I Tear Sheet -m-.20,1971 countries had not provided enough qualified local nationals for training: . (See p. 21.) Other problems were: r L --The objectives agreed upon for one project were not achieved because . the host government unilaterally changed the project. (See p. 26.) --Projects in one country were delayed because of the host government's delays in issuing temporary employment passes which were necessary for AID contract employees to work in that country. (See p. 27.) --AID had difficulty in evaluating the performance of its contract employees at the executive level and in ensuring that they were used to fill only essential positions, because of a lack of information on the activities of these contract employees within the host government. (See p. 27.) AID has worked, especially since 1967, to improve its technical assis- tance evaluation system and to overcome other problems hindering the ef- fectiveness of its efforts. (See p. 31.) AID said that, since GAO's fieldwork, several other changes have been made, including further im- provements in the technical assistance evaluation system. RECOMMENDATION The Administrator, AID, should obtain an independent evaluation of AID's current practices and procedures and determine the extent to which the I types of weaknesses pointed up in this report have been overcome in actual practice. AGENCYACTIONSAND UNRESOLVED ISSUES The Assistant Administrator for Africa recognized the need to ensure that the requirements for measurement criteria were incorporated into both the project planning and programming documents and the technical service con- tracts. GAO was advised that improved techniques and procedures were being utilized for new projects. The Assistant Administrator agreed with GAO on the need to deal with the problems of contract staffing and inadequate numbers of local nationals for training. The extent to which these matters have been corrected in actual practice has not been determined. I I Contents Page DIGEST 1 CHAPTER 1 INTRODUCTION 3 2 MEASUREMENT OF CONTRACTORPERFORMANCE AND PROJECTRESULTS 6 Higher Agricultural Education project 7 Crop and Livestock Development Extension project 11 Range Development project 15 Other projects 16 3 DIFFICULTIES IN FILLING CONTRACTPOSITIONS 19 4 PROBLEMSOF HOST COUNTRYSUPPORT 21 Training of host country nationals 21 Not enough candidates selected for training 22 International transportation expenses 24 Dissolution of the University of East Africa 24 Assignment of trained participants 25 Unilateral decisions by host government ministries 26 Temporary employment passes to contract employees 27 Use of contract employees in host country positions 27 5 ACTIONS TO IMPROVETECHNICAL ASSISTANCE EFFECTIVENESS 31 6 CONCLUSIONSAND RECOMMENDATION 41 Recommendation to the Administrator, AID 42 7 SUMMARY OF AID COMMENTS 44 APPENDIX Pane I Letter dated May 7, 1971, from the Auditor General, Agency for International Develop- ment, to the General Accounting Office 47 ABBREVIATIONS AID Agency for International Development GAO General Accounting Office GENER$L ACCOUNTING OFFICE CRITERIA NEEDED FOR MEASURING TECHNICAL *REPORT 70 THE ADMINISTRATOR ASSISTANCERESULTS AND CONTRACTOR AGENCY FOR INTERNATIONAL PERFORMANCE DEVELOPMENT Agency for International Development Department of State B-160789 DIGEST -_--_- WHY THE REVIEW WAS M4DE Contracts are an important instrument of the Agency for International Development (AID) in providing technical assistance. As of December 31, 1970, AID had 1,191 contracts outstanding totaling about $679 million for technical assistance in 58 countries. Because of the dollar amounts expended and the extensive use of this means of providing technical assistance, the General Accounting Office (GAO) examined AID's use of technical service contracts for 10 agricul- tural projects in Kenya, Tanzania, and Uganda. In carrying out technical assistance projects, AID contracts with educa- tional institutions and qualified firms, associations, and individuals. U.S. universities have been among AID's primary sources. FINDINGS AND CONCLUSIONS Neither AID's project planning and programming documents nor the contracts themselves provided specific targets, goals, or objectives. AID therefore could not evaluate progress and contractor performance objectively. Notwithstanding the absence of objective measurement criteria, some positive and significant results have been achieved through AID’s technical service contracting. For example: --AID was instrumental in introducing vocational agricultural education in Kenya's secondary schools. (See p. 16.) --AID assisted the Government of Uganda to build and expand a strong and independent cooperative movement, particularly in the,dairy industry and in livestock ranching. (See p. 16.) Various problems, however, have hindered the effectiveness of AID’s efforts. There were delays in recruiting and placing contract employees in-country and some employees appeared to lack desired qualifications. (See p. 19.) Other problems limiting the effectiveness of technical assistance were re- lated to host country actions. For example, preparing host country na- tionals as replacements for contract employees is an important objective of the projects. This objective was not fully achieved because host 1 countries had not provided enough qualified local nationals for training. (See p. 21.) Other problems were: , 1) --The objectives agreed upon for one project were not achieved because the host government unilaterally changed the project. (See p. 26.) *_ --Projects in one country were delayed because of the host government's delays in issuing temporary employment passes which were necessary for AID contract employees to work in that country. (See p. 27.) --AID had difficulty in evaluating the performance of its contract employees at the executive level and in ensuring that they were used to fill only essential positions, because of a lack of information on the activities of these contract employees within the host government. (See p. 27.) AID has worked, especially since 1967, to improve its technical assis- tance evaluation system and to overcome other problems hindering the ef- fectiveness of its efforts. (See p. 31.) AID said that, since GAO's fieldwork, several other changes have been made, including further im- provements in the technical assistance evaluation system. RECOMMENDATION The Administrator, AID, should obtain an independent evaluation of AID's current practices and procedures and determine the extent to which the types of weaknesses pointed up in this report have been overcome in actual practice. AGENCYACTIONSAND UNRESOLVED ISSUES The Assistant Administrator for Africa recognized the need to ensure that the requirements for measurement criteria were incorporated into both the project planning and programming documents and the technical service con- tracts. GAO was advised that improved techniques and procedures were being utilized for new projects. The Assistant Administrator agreed with GAO on the need to deal with the problems of contract staffing and inadequate numbers of local nationals for training. The extent to which these matters have been corrected in actual practice has not been determined. 2 CHAPTER1 INTRODUCTION Technical assistance to less developed countries covers the full range of planning, analytical, educational, advi- sory, and other activities traditionally devoted to the en- hancement of human resources and institutional development. The Agency for International Development provides technical assistance, under the foreign aid program, through technical service contracts with educational institutions and techni- cally qualified firms, associations, and individuals. In many instances such contracts make up a significant part of technical assistance projects. As of December 31, 1970, AID had 1,191 technical ser- vice contracts totaling about $679 million for work in 58 countries. About 133 of these contracts, totaling about $105 million, were for technical assistance to 19 African countries. Because of the millions of dollars of foreign aid funds expended by AID through technical service contracting, the General Accounting Office examined- into AID's use of technical service contracts to implement technical assis- tance projects. Our review was complicated by the large number of small dollar amount contracts, the diversity of specific contract purpose, and the wide geographic disper- sion of contract work. After considering these factors we reviewed 10 projects, primarily for education in the agri- cultural sector, in the east African countries of Kenya, Tanzania and Uganda. AID's assistance to these countries which constitute the East African Community is heavily con- centrated in agriculture, education, and the development of community infrastructure and services. The projects reviewed and AID's expenditures were as follows: . AID expenditures as of March 31, 1971 Contract Project portion (000 omitted) Kenya: Crop and Livestock Extension $ 2,313 $ 319 Higher Agricultural Educa- tion 1,480 1,200 Nairobi Veterinary Faculty 1,839 1,047 Range Development 900 314 Vocational Agriculture Edu- cation 857 686 Tanzania: Agricultural College 1,340 953 Public Service Staff Support 1,167 1,107 Uganda: Agricultural Education 2,239 1,674 11 Cooperatives 2,552 988 Makerere Agriculture Faculty 1,199 1,079 Total $15,886 $9,367 AID started these projects in 1964 or earlier and some of them are not expected to be completed until 1974. We observed some accomplishments from the projects and several problems which hampered contract and project effec- tiveness. The problems identified in our review appear to be symptomatic of problems that will require AID's continu- ing attention. Our report highlights some of the accom- plishments of these particular projects and some of AID's problems in implementing technical assistance projects and in using technical service contracts. Project documentation at AID/Washington and at AID's missions in east Africa was reviewed, and relevant matters were discussed with AID, contractor, and host country offi- cials. Our work in the three east African countries was 4 . I done during April through July 1970, and subsequent inqui- * ries were made at AID/Washington. A draft of our proposed report was submitted to AID for comment on March 5, 1971. Comments of the Assistant Administrator for the Bureau of Africa were received on May 7, 1971, and have been recognized, as appropriate, in the body of this report. A copy of the complete comments is included as the appendix. CHAPTER2 MEASUREMENT OF CONTRACTOR PERFORMANCE AND PROJECTRESULTS Clearly defined project and contract targets, goals, and objectives are essential to measuring anticipated proj- ect results and evaluating contractor performance, Even though positive results were apparent from the technical as- sistance projects in east Africa, these results could not be objectively-evaluated nor related to anticipated results because of the absence, in the project planning and program- ming documentation, of predetermined targets, goals, and objectives i,n measurable terms. Just as the project results could not be\,objectively evaluated, neither could the con- tractors' performances be objectively evaluated because the technical service contracts were written in the same general terms as the project documents. Our general impression was that some positive results had been achieved through the use of technical service con-. tracts on the 10 projects that we reviewed. For example, AID was instrumental, through a contract with a U.S. univer- sity, in the introduction of vocational agriculture in Kenya's secondary schools. AID also assisted the Government of Uganda, by providing advisors under technical service X contracts, to make considerable progress in building and expanding a strong and independent agricultural cooperative movement. There appeared to be positive results from the projects, but these results could not be compared with the anticipated results because of the absence of quantitative criteria in the supporting project documents. The project planning and programming documents, for the most part, were general; they lacked specifics as to the targets, goals, and overall objectives of the particular projects. Therefore there were few bench marks with which to compare project results or to measure progress. These factors severely limit the objective evaluation of the ef- fectiveness with which foreign aid funds are being used to provide technical assistance. 6 * The failure to clearly specify in the project documents the plan of action and anticipated accomplishments adversely affected the implementing technical service contracts, The contracts contained almost identical statements of overall goals and objectives as those in the project documents. As the purpose of these contracts was to provide technical as- sistance, the lack of measurement criteria raised the ques- tion of what the contractors were to accomplish by their advice and assistance. Our observations as to some of the positive tangible results and some of the measurement problems are highlighted in the following sections. HIGHER AGRICULTURALEDUCATIONPROJECT The objective of this project, as recorded in the agree- ment between AID and the Government of Kenya on May 29, 1962, was to assist Egerton Agricultural College at Njoro, Kenya, to strengthen its resources and capabilities as an agricul- tural college providing training at the diploma level. Specific project activity goals were listed in three areas of activity and were stated in general terms. These stated goals were to assist the college in: "1. Planning and implementing educational and administrative policies aimed at improving in- struction standards; selecting fields of emphasis necessary in meeting Kenya's expanding training needs relating to agriculture; and expanding the training operations of this college, "2. Strengthening existing faculty resources and developing potential faculty members to meet increasing training responsibilities as the Col- lege broadens its curriculum and expands its stu- dent enrollment. "3. Obtaining certain commodity items nec- essary in carrying out its education and train- ing responsibilities." 7 The objective of the technical service contract, which was signed with a U.S. university on September 25, 1962, was . the same as the project objective; that is, to provide tech- nical advice and assistance to Egerton Agricultural College to strengthen its resources and capabilities as an agricul- tural college providing training at the diploma level. The following examples are illustrative of the seven advice and assistance services to be provided by the contractor. "A. Advice and assistance in the training of persons in animal husbandry, This will in- clude advice and assistance in the development and conduct of improved courses in this field, Courses will include those given to students tak- ing the regular training provided by the College and refresher courses for field agricultural ex- tension personnel. 6 "B. Advice and assistance to the College in developing and conducting courses emphasizing improved agricultural extension techniques and methods which can be applied in the Cooperating Country." The original contract was superseded on June 30, 1965, by a new contract. Although it has been amended several times, the contract objectives and the services to be pro- vided have not been made more specific. The two services quoted above from the superseded contract are also illustra- tive of the services to be provided under the new contract. Both the project documents and the contracts lacked specific time targets, anticipated project accomplishments to meet specific time targets, or the anticipated project completion situations. Lacking also were criteria by which to determine overall attainment of the objective of increas- ing agricultural productivity. AID stated that the overall performance of the higher agricultural education project was satisfactory, even though anticipated project results were not quantified. An ap- praisal report on the project prepared in August 1969 stated that "Anticipated results are difficult to quantify, Proj- ect costs are investments in trained manpower for the agri- cultural sector of Kenya." 8 . The results of this project cannot be compared with those anticipated when the project was started, but there have been positive results. This project helped the college change from a 2-year institution to an institution offering 3-year diploma courses. It improved teaching techniques and practical training and introduced the U.S. system of student evalua- tion. Contract team members developed specialized fields of study leading to diplomas in agricultural engineering, range management, and agricultural education. Enrollment in- creased from about 150 in 1962 to over 600 in 1970, The college is now reportedly able to provide qualified persons for positions within the Government of Kenya and for govern- ment positions in other African countries. From our review of the records, it appears that, during the early years of the contract, there was a lack of direc- tion or specific project purpose. We believe that the prep- aration and use of a work plan would have been an aid in more quickly defining the course of action to be followed by the contractor. The original contract for this project required the contractor to develop a detailed work pian within 120 days after arrival in-country, subject to subsequent revisions as considered necessary by AID or the contractor. After more than 1 year in Kenya, the contractor had not furnished work plans. There was still no work plan for the project in the records at the time of our review. The lack of such a nec- essary planning and implementation document inhibits the identification of problems and makes it relatively impossible to monitor progress, especially for projects such as this where specific targets, goals, and objectives have not been previously identified. 9 April 1970. Construction of a new laboratory for the agriculture engineering department at Egerton College. The building was designed by the AID contract team at substantial savings to the college. Further savings were achieved by the use of student and local labor. 10 CROPAND LIVESTOCK DEVELOPMENT EXTENSIONPROJECT The Crop and Livestock Development project was started about 1955, but AID advisors did not participate until 1962. The project consisted of several subprojects, with the objec- tive of assisting the Government of Kenya to move farmers into the cash economy and to increase and improve farm pro- ductivity. A project agreement with the Government of Kenya on May 9, 1966, for continuation of the agriculture extension element of the project specified that AID would provide three contract rural development area advisors. These advisors were to advise the provincial agriculture officers and their staffs in fulfilling the National Agriculture Development Plan as it applied to the provinces to which they were as- signed. The three advisors were provided to the Government of Kenya under an AID contract of August 1966 with a nonprofit organization. The contract objective was for the contractor to provide the Government with technical advice and assis- tance in its efforts to move the African farmer into the cash economy and to increase and improve farm productivity by: --Improving the effectiveness of the African farmer through advanced technical advisor services. --Establishing sound agricultural institutions that were interrelated and directed toward social and economic development. --Utilizing the pastoral areas more efficiently to in==. crease their contribution to the overall economy. --Developing land resources and better methods of culti- vation. m-Promoting greater utilization of the inland lakes and the sea as sources of food and export earnings. The contract advisors' primary responsibilities were to assist in fulfilling the National Agricultural Development Plan as it applied in the provinces where they were assigned. 11 These responsibilities were stated to entail working with, advising, and assisting Kenyan officials in six areas, such * as: "Working with the Provincial Agricultural Officer in developing a more efficient overall rural de- velopment program and obtaining the cooperation of all associated government agencies in the province on the implementation of such program." * * * * "Advising and assisting the district officials and agricultural workers on methods and tech- niques of implementing improved agricultural practice demonstrations such as the growing of maize, sugar cane, sisal, peanuts, forage crops and livestock improvement." Positive tangible results were achieved by one of the advisors in irrigation activities and in pollination by use of bees. The advisor planned and promoted an irrigation scheme at one village in the Coast Province of Kenya whereby two rice crops a year could be produced, using water pumped from the Tana River. The village council agreed to assume responsibility for the project, to furnish funds to develop the fields, and to provide money for the maintenance and op- eration of the pump and engine supplied by AID under a self- help program. The advisor supervised surveying the fields, installing the pump and engine, preparing the seed beds and plant nursery, and transplanting the rice shoots to the fields. Without irrigation, the farmers had harvested the equivalent of one and a half crops during the previous 5-year period and the community had been frequently on famine relief. The advisor estimated that the irrigation system now provided the capability to harvest 10 crops in a 5-year period. The success of the project encouraged two other villages on the Tana River to take steps to establish irriga- tion systems, with the advisor acting only as a consultant to the Government of Kenya's district agriculture officer. Thus the first irrigation project demonstrated that increased crop production was possible and provided the African farmer 12 ---. Credit AID rural development area advisor September 1968. Just moments after the engine was started for the first time, villagers in the Oda settlement express their pleasure with the pump, supplied by AID, which they helped to install and which will bring water from the nearby Tana River to their rice fields. This irrigation project was planned, organized, and implemented by the AID agricultural advisor assigned to the area’s provincial agriculture office. with an incentive to raise his standard of living by chang- ing and improving his methods of farming. Soon after his arrival the advisor observed that the production of mangoes, cashew nuts, and other crops requiring pollination was very low, considering the size and condition of the trees. After a study revealed that few bees lived in the area, the advisor embarked on a program to establish beekeeping as an active project. Hives, equipment, and bees have been exhibited at local fairs and agricultural shows to impress the local people with 13 the importance of beekeeping. Lectures on modern beekeeping methods have been given at mleetings of young farmer groups, . at farmer-training centers' classes, and at seminars for secondary school science teachers. The advisor, who currently has 35 swarms of bees at various locations in the coastal area, stated that the pro- duction of certain crops in these areas had increased, Cer- tain government officials feel the work to date justifies further study, and the head of the Animal Husbandry Division of the Kenya Ministry of Agriculture has expressed an inter- est in establishing an apiculture section within the Minis- try. A third country has discussed with the Government the possibility of providing funds and training for the new sec- tion. A June 1969 AID appraisal report stated that overall performance of the project had been satisfactory in advancing the project goals. The August 1969 appraisal report, how- ever, stated: "No reliable data are available to indicate spe- cifically the extent to which the agricultural extension service has contributed to the increase in crop production, but it is fairly evident that if the projected increase in output is achieved, a very substantial credit must go to the agricul- tural extension service for its part in this ef- fort." (underscoring supplied.) Thus, while this project manifested some positive tangi- ble results, it also lacked specifics in the project and con- tract documents by which to gauge project progress, contrac- tor performance and attainment of the higher overall objec- tive of increasing farm productivity and moving the farmer into the cash economy. We believe that this project empha- sizes the problem and the need for more definitive criteria for evaluating project effectiveness and contractor perfor- mance. 14 . RANGEDEVELOPMENT PROJECT The general objective of the Range Development project was to assist the Government of Kenya to develop its vast range resources, a high priority in Kenya's development plan. The stated specific goals comprised four general areas in which technical advice and assistance were to be provided. Among the AID contributions were three range management advisors. The goals and duties of these advisors were iden- tified in the implementing technical service contract only as general areas in which the advisors were to provide ad- vice and assistance. Examples of these goals and duties were: "Assist in the development of the Range Management Division of the Ministry [of Agriculture and An- imal Husbandry] and provide technical advice to the staff of this Division. "Provide technical advice and assistance in estab- lishing local grazing associations.*' * * * * * "Assist in developing a grassland range manage- ment and bush control program that will provide the necessary range area for increasing the live- stock production program." One of the advisors said that there was very little re- semblance between the job described to him by the organiza- tion with which AID contracted and the job he was actually performing. Another advisor said that his instructions were simply to assist the Government of Kenya range officer. The third advisor said that the job description given to him was vague and incomplete. As a result the advisors had diffi- culties in determining the specific activities toward which to direct their efforts. The scope of services in the contract was cited by the advisors as being particularly unrealistic. The advisors were to assist in establishing about 2,000 ranch units and grazing schemes in three provinces in Kenya. This number 15 was believed to be unrealistic in view of the small number that had been established, the poor prospects for establish- . ing a very large number in the future, and the lack of plans for establishing a larger number. As of April 1970, 60 ranch units had been organized. The work plan for the coming 2-year period called for establishing an additional 102 units. If 162 ranch units are organized by April 1972, the anticipated project completion date, only 8 percent of the total projected ranch units will have been established. ‘OTHER PROJECTS There have been positive tangible results on other projects even though measurement of project progress and ef- fectiveness was a problem. For example, through the Voca- tional Agriculture Education project, AID inaugurated the teaching of agriculture in the Kenya secondary schools in 1960 with a pilot project at a school in the Western Prov- ince. The apparent success of the pilot project led to the award of a university contract under which instructors were provided for six additional schools. The project has been expanded into a program to include 25 schools--seven spon- sored by the United States, 13 by the World Bank, and five by the Government of Kenya. The Government of Kenya plans to expand this program into additional schools as funds and teachers become avaiable and to introduce a vocational agri- cultural course into the primary schools. Since 1963 AID has provided, through the Agricultural Cooperatives project, from three to six contract staff mem- bers to assist the Government of Uganda to build and expand a strong independent cooperative movement. Project activi- ties were directed primarily to improving management skills through training within the cooperatives and the responsible government ministry, assisting new cooperative groups in working with new and diversified crops, and establishing re- lations with credit and marketing organizations. The movement has shown considerable progress. Since 1963 the number of cooperatives has increased from 1,700 to 2,500 and membership has increased about 60 percent. The project documents indicate that cooperatives are now widely accepted as important instruments in teaching better business, better farming, and better living at the country's local so- cial level. 16 The dairy industry and livestock ranches are two areas where the cooperative movement has shown progress. In 1965 there were three cooperative dairy groups in Uganda market- ing 150 gallons of milk daily. In contrast, 19 groups were marketing 4.,000 gallons of milk daily in 1969. These groups vary in size from small collection centers processing 50 to 100 gallons of pasteurized milk daily, to one large coopera- tive processing over 700 gallons of pasteurized milk daily. This large cooperative was organized in 1966 and con- tinues to grow and prosper. It delivers fresh milk to all principal markets in the district. Income of the coopera- tive has averaged over $10,000 a year since it was organized 4 years ago and has been used to construct a new dairy plant and to purchase new pasteurization equipment. The coopera- tive serves as a model and demonstrates what can be achieved by milk producers who own and operate their own businesses. The 20 cooperative livestock ranches that have been organized since 1965 in Uganda have increased their cattle from 6,000 head valued at $350,000 during 1965 to over 12,000 head valued at $850,000 during 1969. On these ranches were placed 2,400 head of imported purebred females. Plans have been made to establish another 25 cooperative ranches during the next 2 years. The capital required to finance these ranches will be supplied by the new ranch members and by credit obtained through local commercial banks. Thus there have been apparent positive tangible results from this project as with other projects, but the same prob- lems continue as to measurement of progress toward goals and overall objectives. The project documents and the technical service contracts contain very few quantitative indicators by which to measure progress or intermediate targets and goals or effectiveness of AID's contribution to the cooper- ative effort. 17 April 1970. The vocational agriculture teacher at Njoro Secondary School, Njoro, Kenya, admires the young bean crop under cultivation by the students. A substantial por- tion of the food needed for about 300 boarding students is grown under the vocational ag- riculture program on the 16-acre school farm. 18 CHAPTER3 DIFFICULTIES IN FILLING CONTRACTPOSITIONS Various contract staffing problems were common to the 10 projects and related contracts that we reviewed. The primary problem was delay in placing contract employees on the job in-country. Other, and perhaps related, problems were employee turnover and employee qualifications. The direct effect of these problems was not readily apparent in all instances; however, it appeared that they had contrib- uted to delays in achieving project goals and to extensions of project completion dates. An example of delay in placing contract employees in- country is found at the Crop and Livestock project in Kenya. On this project the contractor did not provide the required three rural development advisors until 5 months, 6 months, and 2-l/2 years after the contract had been signed in 1966. Another example is found at the Makerere Agriculture Faculty project in Uganda where the contractor's chief of party ar- rived 6-l/2 months after the position was authorized, and two of three staff members needed by June 1966 to prepare for the academic year arrived in June 1967. These delays in placing contract employees on the job may be due to a combination of reasons. Some of the appar- ent reasons were delays by the contractor in recruiting the required employees and delays by a host government in issuing temporary work permits which were necessary for contract employees to work in that country. AID contracts usually require contract employees to serve at least a 2-year tour. A contract technician needs about 6 months to become acclimated overseas, according to AID, and probably another year to demonstrate real, visible progress in work objectives. Thus the effectiveness of con- tract employees was limited, particularly during the early phases of project operations, because they were new on the job. Another factor limiting the effectiveness of contract employees was the absence of an overlap of tours between departing and arriving employees to provide continuity of experienced employees on the projects. 19 On some projects it appeared that the employees pro- * vi&d by the contractor had less than the desired qualifi-' cations. It was not readily apparent whether this resulted - from the absence of specified minimal education and experi- ence requirements in the contracts or whether employees with the desired qualifications could not be obtained. AID planning and programming documents have indicated for example, that certain positions should be filled by se- nior staff members from a given university's faculty, but the contracts listed only the title of the positions to be filled, such as lecturer in animal husbandry, and did not prescribe minimal qualifications for a college lecturer in this field. Many persons recruited by a contractor for teaching positions at the agricultural colleges in Tanzania and Uganda had never taught at the college level. Some had been only recently graduated from college, and others had been teaching in high schools or working in state agricultural extension programs or at research stations. The contract university for the Agricultural College project in Tanzania employed a chief of party whose main duties consisted of college administration and development, but whose professional experience had been limited to U.S. county and regional agricultural extension work. 20 1 , . CHAPTER4 PROBLEMSOF HOST COUNTRYSUPPORT Various difficulties associated with host country ac- tions and project support have hindered progress on the AID technical assistance projects in east Africa. The most se- rious problem was the failure to provide for training an adequate number of host country nationals to replace AID contract employees and to carry on and expand the institu- tions and techniques initially sponsored under foreign as- sistance. Other problems included the adverse effect on AID- financed projects of a host government's unilateral deci- sions, delays by a host government in issuing the employment passes required before contract employees could work in- country, the lack of assurance that a host government used AID-financed executive employees to fill only essential posi- tions, and the lack of information on the work performance of the AID-financed executive employees. TRAINING OF HOST COUNTRYNATIONALS A principal purpose of technical assistance is to pre- pare the people of the host country to carry on and expand the institutions and techniques initially sponsored under foreign assistance. We noted several problems limiting the achievement of this purpose on the PO projects in Kenya, Tanzania, and Uganda, The primary problem was the nonavail- ability to the AID projects of qualified nationals to be trained. Other contributing problems were a prolonged disagree- ment between AID and the three governments over payment of expenses for the trainees a transportation to the United States and a reduction in the number of possible trainees for specific projects because of the dissolution of the re- gional University of East Africa. Another factor limiting the achievement of specific project objectives was the as- signment of trained nationals to projects other than those for which they were trained. 21 . Not enough candidates selected for training Prior to the agreements with the host governments for these projects, neither the missions nor the host govern- ments determined that an adequate number of participants would be available for training and for replacing AID con- tract employees. Only after many of the projects were well under way was it apparent that not enough participants would be trained within the period required by the project sched- ules. The Vocational Agriculture Education project in Kenya is one example of the participant training problem. When this project was started in 1964, AID stated that it would be necessary to train locally an adequate number of teachers to support future expansion of the program. In spite of this recognition of need, training plans were not made until late in 1968 when the AID Mission in Kenya received a re- quest from the Government of Kenya for a crash program to train 30 agriculturalists as secondary teachers. The program called for the training of two groups of 15 students each for 22 weeks to meet the expanding vocational agricultural education program. Training was to start in January 1969 for the first group and in June 1969 for the second group. Shortly before the training program was to start, the Government of Kenya advised the Mission that the Government could provide only 10 candidates for the January class. In commenting on the first group of potential teachers, a U.S. contract staff member identified the following problems. --A week after the scheduled starting date for the training program, less than half of the required num- ber of students were present. --After 5 weeks of training, only nine students were enrolled, three of whom had no formal agriculture training. --Because of the late arrivals of students and the dif- ferences in their backgrounds and training, the course content had to be modified and the quality of train- ing was lowered. 22 --Because the students had been selected without their consent, they had a poor attitude toward learning. A decision was made to cancel the second class, even though the Government of Kenya advised that sufficient can- didates were available. These reductions in the teacher training program contributed to the July 1970 shortage of eight teachers for the 25 secondary schools in the voca- tional agriculture program. One school had no agriculture teacher, and the course was being conducted by other teach- ers. At seven schools the workload reportedly was too heavy for the assigned teachers. These vacancies were ex- pected to exist until the December 1970 graduates from Eger- ton Agricultural College became available to fill the posi- tions. The inadequacy of the number of participants available for training on AID's Public Service Staff Support project in Tanzania is discussed in a following section of this re- port on the use of contract staff in host country positions. On another project in Tanzania, the Agricultural College project, an AID evaluation report of January 1970 states that participant training has proceeded at a slow pace, due mainly to the continuing unavailability of qualified Tanza- nian candidates for training. This report states further that the slow pace of this training has resulted in inade- quate staffing at Morogoro College in Tanzania and has con- stituted a delay in achieving the target of training for staffing the college, Nonavailability of local nationals or participants for training was a problem in Uganda as it was in Kenya and Tan- zania. For example, one of the goals of the Makerere Agri- culture Faculty project in Kampala, Uganda was to train re- placements for the eight AID contract staff members assigned to Makerere. The contract is scheduled to terminate in 1971. Even though the training program was conceived in 1966 and a number of years is required to provide the necessary educa- tion and on-the-job training for a replacement, only three participants had been identified and were receiving training as of 1970. 23 International transportation expenses In 1968 AID asked the three east African governments to pay the transportation costs for sending their partici- pants to the United States for training. In AID's view the payment of transportation costs would be a concrete demon- stration of self-help by the east African governments, would be an encouragement for more careful selection of candidates and better utilization of their skills after training, and would be consistent with cost-sharing policies in other geo- graphical areas. Tanzania and Uganda agreed in 1969 to pay a portion of the transportation costs. One participant's training was delayed more than 6 months before the agreement was nego- tiated, and another participant's training was delayed sev- eral months because Uganda did not adhere to the terms of the agreement, Kenya did not sign an agreement with AID to pay trans- portation expenses of participants until June 1970. During the intervening Z-year impasse about the payment of trans- portation expenses in Kenya, only 21 participants were nomi- nated for the about 80 positions available for training in the United States. Training for these 21 was delayed from 6 months to 1 year. Dissolution of the University of East Africa In the early 1960's, the three east African governments agreed to consolidate the administration of the higher edu- cational institutions in their respective countries under a single regional University of East Africa. Under this unified university system, AID has been pro- viding technical assistance to the Veterinary College in Nairobi, Kenya, and the Makerere Faculty of Agriculture in Kampala, Uganda-- the only institutions in east Africa offer- ing degrees in veterinary sciences and agriculture, respec- tively. Citizens of the entire area may serve as faculty members or may be admitted as students at these colleges. Recently the governments decided to dissolve the Univer- sity of East Africa. Kenya is establishing its own 24 . -, . L . r'agricultural degree program, and Uganda is considering es- . tablishing its own veterinary degree program. For that rea- son the administrators of the Veterinary College and the Makerere Faculty of Agriculture are reluctant to nominate any but citizens of their respective countries for advanced training in the United States under the participant training 0: . portion of the technical assistance agreements. The administrators believe that, when the unified sys- tems have been dissolved, the participants will be under strong pressures to take positions on faculties in their own countries after their American university training. Thus the student base from which to select qualified candi- dates for advanced degree training in the United States has been narrowed greatly. There are other problems of host country support, in addition to the difficulties encountered in securing partic- ipants for the eventual replacement of AID-financed staff members. For example, officials at the veterinary and agri- cultural schools are concerned about the loss of present faculty members under the separate-university concept. The Makerere Faculty of Agriculture already has lost one member, and the Veterinary College expects to lose six or seven of its present staff of 53 members if a separate veterinary faculty is established in Uganda. Assinnment of trained participants Several participants trained in the United States on educational assistance agreements were not assigned to teach- ing positions when they returned. AID documents show that the loss of these participants from agreed teaching positions has been a contributing factor in the failure of certain projects to meet their scheduled completion dates. For example, AID's Higher Agricultural Education proj- ect in Kenya included the training of east Africans for the faculty at Egerton College. By May 1970, 21 of 22 partici- pants that had been sent to the United States for advanced training had returned but only 12 of the 21 were assigned to the Egerton faculty. The diversion of trained partici- pants to other government positions was cited by AID auditors as a contributing factor in the need to extend the project beyond its scheduled completion date. 25 TJNIWrERAL DECISIONS BY HOST GJVERNMENT MINISTRIES A unilateral decision by the Ministry of Agriculture in Uganda seriously impaired the success of AID's Higher Agricultural Education project in that country. The primary purpose of the project was to meet the estimated manpower needs of Uganda's agricultural extension service by provid- ing assistance to two agriculture colleges in Uganda for training junior and senior extension workers. The agreed educational goals of the project were to train two senior officers in a 3-year-diploma course for every three junior assistants trained in a Z-year- certificate course. Even though there is only a l-year dif- ference in education, there is considerable difference be- tween the junior and senior positions. The junior assis- tants work more closely with farmers at the local level and are assigned less territory to work. The senior assistants are considered to be administrative and they supervise one or more junior assistants. The salaries of the senior as- sistants are about twice those of the junior assistants. Apparently wanting to Africanize positions more rapidly, the Ministry unilaterally decided to eliminate the certifi- cate program from these schools and to offer the diploma course only. Staffing in the extension service then became seriously out of balance, and there were nearly as many senior officers in the field as junior assistants. On the basis of its manpower survey published subsequently, how- ever, the Government of Uganda recognized its shortage of junior extension workers and reintroduced the junior certif- icate course in the schools. In the meantime several years were lost in attaining the project objectives. The Mission director informed us that assistance had been continued in this case because the Mission believed that the Ugandan economy could absorb all levels of manpower in the agricultural field. 26 . TEMPORARYEMPLOYMENTPASSES TO CONTRACTEMPLOYEES Until issued a pass a foreigner may not work in Kenya. The Government of Kenya has held up work on some AID proj- ects by delays in issuing employment passes for contract em- ployees. In one case an AID contractor lost a candidate for a position on a Kenyan contract when the candidate cancelled his agreement after waiting, unsuccessfully, for 5 months for a pass to work in that country. This, in turn, created additional delays to recruit another candidate for the posi- tion and to obtain an employment pass for him. There was a 4-month delay in obtaining an employment pass for the new candidate. Consequently a total of over 2 years elapsed be- fore the contractor was able to place an employee on the job. The Government of Kenya delayed issuing a temporary em- ployment pass for 5 months for another contract employee and thus delayed the starting of work on organizing courses for a specialized field of agricultural education.1 The AID Mission on occasion has contacted the Govern- ment of Kenya regarding delays. USE OF CONTRACTEMPLOYEESIN HOST COUNTRYPOSITIONS A type of technical assistance is the financing of op- erational employees at the executive level to fill respon- sible positions within the host government. Under this plan, the host government pays the basic national salary and AID supplements it for a total salary comparable to U.S. standards. The Public Service Staff Support project in Tan- zania is of this type. AID has had some problems in con- trolling this project, particularly in ensuring that AID's 1 In a comment dated May 5, 1971, AID stated that "There have been no instances of delays cited since the GAO team visited Nairobi." 27 contract employees are used to fill only essential posi- . tions, in getting nationals trained and assigned to replace contract employees,and in obtaining information on the work performance of contract employees. The project agreement, dated May 1964, between AID and the host country provided, in part, the following project description. "The purpose of this project is to assist the United Republic of Tanganyika and Zanzibar (Tanzania) to meet its short-term high level manpower requirements in order that the essen- tial services and projects as articulated in the Five Year Plan can be met. There currently exists a number of active programs which will narrow if not close the high-level manpower gap by the end of the Five Year Plan period. Until this gap is closed, however, a severe shortage of trained personnel will continue to exist in both the senior administrative and professional areas. "In order to assist in the rapid upgrading or training of local personnel for those positions supplied through this agreement, it is explic- itly understood that at least one citizen of the United Republic will be identified and, where possible, trained to take over the job be- ing done by the U.S.A.I.D. supplied expatriate person at the earliest possible time." By June 1970 AID had provided 28 contract executive employees for the Tanzanian Government under the Public Ser- vice Staff Support project. Nine of these, or about 32 per- cent of the total, returned to the United States after serv- ing their tours of generally 2 years each and without having trained or having been replaced by a Tanzanian. Another six of these contract executives were in their second 2-year tour before local nationals were designated as replacements. In one case a Tanzanian had been selected for on-the- job training to qualify as a replacement but had not yet be- gun this training even though the contract executive had . . served 1 year of his second tour. In other cases Tanzanians being trained by their contract counterparts were not, for various reasons, expected to be qualified as replacements for the contract employees by the end of the contract em- ployees' tours of duty. Because of the need to give more attention to identi- fying and training host country employees to replace AID op- erational employees, AID issued a circular in April 1969, calling attention to existing guidance issued in January 1968. This circular stated that each host government should be able to develop a plan, prior to project consideration and approval, setting forth in specific detail the measures to ensure development of replacement employees within a care- fully estimated time period. The Public Service Staff Support project was terminated as of June 30, 1969, except for those contract employees then authorized. Contract employees to be provided after July 1, 1969, were to be incorporated into separate project agreements by fields of activity. According to AID this would enable a project manager to determine the relevance of requests for recruitments and to malce certain that (1) each approved request was related to U.S. project objectives, (2) each approved request had a high priority in the Tanza- nian development plan, and (3) a qualified Tanzanian replace- ment could be identified and trained. AID employees in Tanzania told us that, under the new procedure, the Mission had adopted the practice of sending a Mission official to visit the requesting offices of the Tanzanian Government. The Mission official ascertained the significance of the position by interviewing Tanzanian Cov- ernment officials and by examining organizational charts and other documents. We were told that the Mission, when nec- essary, assisted the requesting office to prepare an ade- quate position description. Since AID contract employees on this program are hold- ing executive positions in the Government of the host coun- try, there is an understandable limit OR AID's monitoring of the employees' activities. Nevertheless AID recognizes that it must have some channel of information on U,S,- financed employees to appraise contract performance in 29 individual cases and to appraise the results and problems * of the program generally. This need for information was met only partially in Tanzania by an agreement in 1967 between the Mission and the Government, requiring contract executive employees to sub- mit a tour-end report through the Tanzanian Government to the Mission after 18 months of their Z-year tours. AID internal auditors have reported on the inadequacy of this single tour-end report from contract employees as a source of information on the employees' activities and per- formances. The auditors expressed their belief that 18 months after starting a 2-year tour was too late for an ef- fective action by AID. Mission officials say that they try to fill the information gap by discussions with Tanzanian Government officials on the activities of contract execu- tives. 30 CHAPTER5 ACTIONS TO IMPROVE TECHNICAL ASSISTANCEEFFECTIVENESS The problems noted during our review are of long stand- ing, and AID has worked toward alleviating these problems and improving the effectiveness of its technical assistance. Recent actions have been taken in two major areas--improving the project programming and monitoring system and implement- ing, on a pilot basis, a new contractual arrangement with U.S. universities. AID expects that the impact of the ac- tions in both areas will result primarily in better plan- ning, programming, and monitoring of technical assistance projects and of implementing contracts. AID will need, how- ever, to monitor continuously technical assistance projects to ensure that its new regulations are implemented effec- tively. In 1967 AID revised the instructions for planning, pro- gramming, and monitoring its technical assistance projects to improve the depth and quality of the planning and evalu- ative processes. The revised instructions for preparing the project planning and authorization document stated that the docu- ment's relationship to the process of evaluation must be carefully considered-- that it must be in terms which would facilitate subsequent evaluation. The instructions further emphasized that the planning document should show what the situation will be upon termination of the project, compared with the beginning, and when termination will occur. Thus considerable emphasis was placed upon clearly identifying during the planning process the targets, goals, and objec- tives in quantifiable terms. A mission project implementing plan was to be prepared, showing in greater detail the relationship between the planned application of resources and the project results. In 1968 AID initiated a project appraisal system that required a comparison of actions taken with actions planned 31 and the identification of unsatisfactory contractor perfor- mance. Effective evaluation under this system was dependent' on the preciseness of program goals and of objectives in the planning documents and contract specifications. AID had difficulty, therefore, in making effective appraisals of projects started prior to about 1967 or 1968, which gener- ally were initiated without specific and meaningful targets, goals, or objectives. Even though AID has revised its instructions, AID has not always provided adequate measurement criteria for re- cently initiated projects, that is, projects started after the planning and programming instructions were revised in 1967 and after the adoption of the project appraisal system in 1968. This was the situation for new projects in Tanza- nia, which, for the most part, replaced the Public Service Staff Support project and the Agricultural College project which were part of the 10 basic projects that we reviewed. The Public Service Staff Support project for providing operational employees at the executive level was terminated in 1969 and was replaced by two other projects, an Agricul- ture Materials and Services project and a Management and Engineering Services project. The Agriculture Materials and Services project was in- tended to assist the Ministry of Agriculture, Food and Co- operatives, in developing the capability for planning, ad- ministering, implementing, and supporting agriculture de- velopment activities. This project was approved by AID/ Washington even though the project documents contained no indication of what would represent completion of the project other than the provision of the specified inputs. Prior to approving the Management and Engineering Ser- vices project, AID/Washington questioned the general nature of the project goals. In this case AID/Washington pointed out the importance of precisely defining the quality and quantity of proposed assistance and, if feasible, of speci- fying a timetable for completion. The project was approved, however, after the Mission in Tanzania advised Washington of the difficulty in expressing precise goals and targets for the project. 32 The Agricultural College project for providing assis- tance to Morogoro Agricultural College was terminated in 1969, but assistance was continuing to be provided through an amendment to the Education Materials and Services proj- ect. After reviewing documentation for the Education Mate- rials and Services project, AID/Washington stated that the overall project objectives were not clear. Although the project objectives were not clarified by the Mission, AID/ Washington subsequently approved the project. Thus, although the need for clear and specific goals and objectives in quantifiable terms had been recognized and emphasized in the AID regulations, these things were not made a part of these three projects, just as they were not made part of the 10 projects which were reviewed by us and which were initiated prior to the 1967 and 1968 revisions to the programming instructions. In 1969 AID contracted for a review of the technical assistance project appraisal system which it initiated in 1968. Although this study was intended to concentrate on the project appraisal system, the project planning, program- ming, and implementation were to be evaluated to the extent that they related to the appraisal system. The contractor reviewed the project appraisal system on the basis of a representative sample of 63 projects. Find- ings on the need for more specifics and evaluative indica- tors in the project planning and programming, similar to those found by us, were reported by the contractor. He stated that ,'*** critical problems confronting AID project evaluation do not arise from imperfections in the PAR [project appraisal report] document and system. Rather, they stem from inadequate proj- ect planning and deficiencies in applying the concept of project management. USAID [Mission] managers lack the logical framework required to define TA [technical assistance] projects in terms that permit objective verification of suc- cess. In the absence of such a framework, useful evaluation--that is, evaluation that produces 33 new project plans with increased confidence in successful completion-- is enormously difficult, if not impossible.tt The contractor reported that improvements in evaluation depended upon resolution of the following three basic prob- lems hampering AID projects. "(1) The purpose of Technical Assistance projects rarely are defined sharply, and the connec- tion between a project and its higher goals is almost never clearly understood by proj- ect staff; "(2) USAID mission staff are not asked to accept explicit responsibility for achieving proj- ect success, as success is highly dependent upon actions of others--thus, there is rarely a clear sense of management responsibility; "(3) Lacking both the orientation that should be provided by clearcut plans and sharply de- fined management responsibilities, and the methodology appropriate to a well-defined experimental situation, the USAID evaluator has found evaluation a difficult task, and has found it particularly difficult to translate evaluation results back into better plans and better projects." The contractor stated that projects had not been clearly related to sector goals. Consequently there appears to be widespread lack of understanding as to why projects are be- ing undertaken-- how they relate to U.S. country objectives and programming goals. Lacking such insight into broader objectives, mission employees find it difficult to intelli- gently replan their projects, and the ability to replan is the key to effectiveness. The contractor's report also discusses an end-of- project status, that is, an anticipation of what should re- sult from the project, stated to facilitate objective veri- fication that the project purpose has been achieved. The importance of objectively verifying achievement of overall 34 project purpose, in addition to measuring specific project outputs, is stressed. The contractor indicates that his view supports that already in the following provision of the AID programming regulations. It 'It is of prime importance, both to the project review and approval process and to the ultimate project evaluation process, that anticipated re- sults of the project be made clear. The concept and specification for "completior2' of the proj- ect *** should be stated with maximum precision.' I1 The contractor recommended that maximum precision in- clude stating how completion of the project, that is, achievement of overall project purpose, will be verified objectively and that the means of verification of project completion be independent of the measurements of project outputs. The contractor's recommended revised evaluation system was built around what he termed 'Ia logical framework for evaluation," that is, showing specifically the relationship between inputs, outputs, project purpose, and sector or programming goal and providing objective criteria in the project planning and programming documentation for measuring projects' outputs and degree of achievement of projects' purposes. On the basis of the contractor's study and recommenda- tions reported in July 1970, AID revised its technical as- sistance evaluation system. Although we have not made a de- tailed analysis of the revised system, it readily is appar- ent that the emphasis of the revised system is on the need for more specifics in technical assistance planning and programming and on building of evaluative criteria into the project design-- needs which were indicated in the projects that we reviewed and those reviewed by the AID contractor. In this respect the AID evaluation handbook which was is- sued as part of the revised system states under the caption of implementation plans: 35 "As life-of-project documents, PROPS[non- ' capital project papers] deal with general strat- egy rather than detailed tactics and schedules. The same is generally true of loan papers, al- though some of them may contain considerable de- tail. In either case, detailed plans of action are needed. These plans also provide the bench- marks for meaningful evaluation of two important aspects-- effectiveness and efficiency. "For non-capital projects, the Joint Proj- ect Implementation Plan (PIP) is prepared in the early stages of the project (see M.O. 1025.2). It sets out the work schedule and certain out- put indicators, as well as such key inputs as personnel, participant and commodity require- ments. The progress of a project toward its established targets can be measured against these output indicators in a quantitative manner. Some projects, such as those of an advisory or institution building nature, do not readily lend themselves to quantitative measures. However, even in these cases, it should be possible to provide some definable steps or forms of be-' havior which are verifiable evidence of achieve- ment," If this renewed emphasis results--in actual practice-- in better project planning and programming, including the building of evaluative criteria into the project design, then AID should be in a position to contract for more spe- cific items of work with a clearer delineation of the con- tractorBs responsibilities which, in turn, would permit a more objective evaluation of contractor performance. In commenting on our draft report, AID representatives stated that: "This installation process has gone forward sys- tematically since the autumn of 1970. Four re- gional training conferences of three days each were held for Evaluation Officers, Program Officers and Deputy Directors of U.S.A.I.D. Missions. These conferences have been followed 36 by one-week training visits to each Mission by teams from Washington to work with Mission per- sonnel in the actual evaluation of two projects per Mission. The evaluations have resulted in action decisions to improve implementation or to modify forward plans. Frequently, the evalua- tions have also been a vehicle for communication with the contractor and the host government which has clarified targets and responsibilities. In order to assist Missions in the conduct of rigor- ous evaluations, AID has issued worksheets and guidelines for project managers and an Evaluation Handbook for Evaluation Officers. Regular train- ing in Washington is also being set up to help overcome the effects of personnel turnover," The impact of better planning, programming, and moni- toring required by the revised system should be felt in all areas of project implementation. The annual evaluation pro- cess should draw management attention to most of the prob- lems disclosed by our review. The performance analysis seg- ment of the evaluation specifically covers the implementing agent's performance, including timely recruiting of U.S. contract employees and training and utilization of a local staff; participant training is evaluated as a separate item, including participant availability and employment after training; various factors relating to host country support are evaluated as separate items. The implementing agents for technical assistance proj- ects are, to a large extent, U.S. educational institutions-- U.S. universities being the implementing agents for six of the LO projects that we reviewed. A joint committee composed of members of the National Association of State Universities and Land Grant Colleges and AID recommended measures, in its January 1970 report, to improve operating arrangements between AID and universi- ties for long-term institution building projects, At the heart of the report is the emphasis that the U.S. univer- sity, AID, and the host country jointly should define goals and operational targets in clear, realistic, and measurable terms, as well as accomplishment strategy, and jointly 37 should perform annual evaluations of progress toward pro-, ' gram goals, AID is testing the committee's proposals on a . pilot basis. The committee report envisioned several phases in the project planning, having the joint participation of all con- cerned, before undertaking a long-term institution building processe Under one phase, reconnaissance and assessment9 a U.S, university under contract with AID develops and dis- cusses the university's views on the validity, feasibility, and best shape of the project. If AID decides to continue with the project, it then extends the contract with the U.S. university for the proj- ect refinement phase. During this phase the U.S. university works with the host institution to define goals and opera- tional targets in clear, realistic and measurable terms; to develop a time-phase work plan, at least for the early part of the project; to develop plans for evaluation, including criteria; and to develop a budget proposal relating incre- ments of financing to increments of the work program, After these phases AID negotiates a long-term operating agreement with the U.S. university. An important aspect of the agreement is a provision for annual joint reviews of progress toward achieving program goals. The report indi- cated that, to a large extent, success of the new arrange- ment would depend upon continuing emphasis on program eval- uation. The comnittee report places considerable emphasis on the efforts that should be made to obtain qualified employ- ees on a timely basis and to train participants and phase them into the project. The proposed new operating arrangement between AID and U.S. universities is primarily for projects for long-term institution building under certain specified conditions and thus is not applicable to all AID contracts with U.S. uni- versities. Even though the new arrangement is primarily for long- term institution building, many of the principles are appli- cable to contractual arrangements for other purposes. This 38 would be especially so for the preliminary planning phases of defining goals and targets in measurable terms and the annual joint program reviews. In another effort to improve its management of techni- cal assistance projects, AID sent a circular in December 1968 to its missions calling attention to a wide range of problems in implementing projects. This circular called attention to how AID's technical assistance project instruc- tions could aid in the technical assistance process; but it emphasized that these instructions became meaningful only when the AID staff was alert to recognizing existing and po- tential problems, was perceptive in ascertaining their causes, was imaginative and resourceful in designing solu- tions, and was objective and firm in taking indicated ac- tion. Thus, as desirable as new procedures and instructions may be, AID recognizes that their ultimate effectiveness de- pends upon the actions of AID employees. In commenting on our draft report, AID representatives stated that: "AID also recognizes that keeping' its personnel alert and effective requires continuing efforts by management at all levels. Thus the December 1968 circular mentioned above has been followed by several other circulars about ways to improve project management. The teams which visited Missions to help install the new program evalua- tion system also conducted training sessions on other aspects of project management besides evaluation. A Handbook for Project Managers has been issued and management training seminars are being given on a regular basis for field project managers. Project Management has been on the agenda of AID's regional Mission Director con- ferences. Despite such continuing efforts, how- ever, it must be recognized that technical as- sistance personnel, either contract or direct- hire, will usually have to overcome local diffi- culties in the environment of the host country. 39 The difficulties which these countries face in ' operating effectively, are among the most im- portant reasons why they need technical assis- tance." CHARTER6 CONCLUSIONSAND RECOMMENDATION The 10 projects that we reviewed were initiated prior to the emphasis in 1967 on building evaluative indicators into the technical assistance project planning and program- ming and clearly specifying what the anticipated results of the projects should be. Because of the absence of quantified goals and objectives in the planning and programming docu- ments, AID was not in a position to make an effective evalu- ation of project results or to compare actual results with anticipated results, The same weaknesses were found in the technical service contracts. Thus AID was not in a position to adequately evaluate the contractors9 performances in pro- viding technical assistance, Similar findings were reported as a result of an AID contractor's review of AID's project evaluation system for 63 projects selected on a representative basis. AID has .worked through the years to improve the effec- tiveness of its technical assistance, including the revising of its technical assistance project appraisal system and its pilot test of new working arrangements with U.S. universities for long-term institution building projects. The greatest impact of these actions, either direct or indirect, appears to be in better planning and programming through clearer identification of project output, project purpose, and project relationship to higher programming or sector goals and through building into the project design evaluative criteria for progress measurement and project re- planning. These actions are necessary prerequisites to more effective use of technical service contracts by a more spe- cific identification of contractors9 scopes of work, tar- gets, and overall contract objectives and by better eval- uation of contractors9 performances in providing technical advice and assistance. Our review, however, of the actions AID took on three projects after incorporation of the foregoing management 41 improvements in its instructions showed that, for those * projects AID had not put the requirements of its instruc- * tions into practice. AID has indicated, as well as has the AID contractor, that it is feasible and practicable to build into the.proj- ects means to evaluate progress and to compare actual re- sults with anticipated results. This is illustrated by the following statement in AID's evaluation handbook published in October 1970 and reprinted in February 1971. The project implementation plan I'*** sets out the work schedule and certain out- put indicators, as well as such key inputs as personnel, participant and commodity require- ments. The progress of a project toward its es- tablished targets can be measured against these output indicators in a quantitative manner. Some projects, such as those of an advisory or institution building nature, do not readily lend themselves to quantitative measures. However, even in these cases, it should be pos- sible to provide some definable steps or forms of behavior which are verifiable evidence of achievement." As AID has indicated that an even greater proportion of projects are to be implemented with contractors, the need now appears to be that of ensuring that the output indica- tors are put into actual practice. RECOMMENDATION TO THE ADMINISTRATOR, AID In our draft report submitted to AID for comment we proposed that AID: 1. Take appropriate action to require in actual prac- tice that the project planning and programming doc- umentation and the technical service contracts con- tain more specific delineations of scope of work, goals, and objectives and quantitative indicators by which progress may be evaluated. 42 . 2. Institute a procedure for measuring on a continuing basis (a) the seriousness and causes of contract staffing problems, including the identification of con- tractors who fail to provide staff on a timely basis and (b) the full significance of the nonavailability for training of host country nationals. We have been advised that, since our fieldwork, AID has revised its technical assistance evaluation system and that "AID has expended a sizeable amount of effort in reorganiz- ing, planning, and establishing documentation systems in or- der to correct deficiencies." We do not know what effect these changes have had in actual practice. We recommend that the Administrator obtain an indepen- dent evaluation of the current practices and procedures and determine the extent to which the types of weaknesses pointed up in this report have been overcome in actual prac- tice. 43 CHAPTER7 SUMMARYOF AID COMMENTS AID's comments on this report (see app.) were directed primarily to the subjects covered by the proposals in the draft report. AID indicated concurrence on our proposal that there was a need to require--in actual practice--the use of more specifics and evaluative indicators in the project planning and programming and in the implementing technical service contracts. AID indicated also that the proposal relating to contract staffing and to training of host country nationals was acceptable. Some elaboration was made upon the inherent difficul- ties in overcoming the problems. AID also highlighted --the difficulty of clearly identifying the effect of AID-financed inputs and outputs because the AID in- puts may be only a small portion of the total ef- fort; --the current AID actions in the process of installing the revised project appraisal system; --the efforts by AID management in keeping employees alert to ways of making technical assistance more effective; --the difficulty in, but the importance of, providing contract employees on a timely basis; and --the situation regarding availability of host country nationals for training. 44 APPENDIX 45 APPENDIX I DEPARTMENT OF STATE AGENCY FOR INTERNATIONAL DEVELOPMENT WASHINGTON. DC. 20523 MAY 7 1971 Mr. Oye V. Stovall Director International Division United States General Accounting Office Washington, D. C. 20548 Dear Mr. Stovall: We have completed our review of the draft report titled "Effective- ness of Technical Service Contracts in Providing U.S. Technical Assistance to East Africa" which was transmitted with your letter of March 5, 1971. Enclosed is a memorandumto the Auditor General from Dr. Samuel C. Adams Jr., Assistant Administrator of the Bureau for Africa, which constitutes the Agency's consolidated response to this report. Sincerely yours, Edward F. Tennant // Auditor General - Enclosure: a/s GAO note: Page numbers and related references in this appendix have been changed to correspond to the pages of this report. 47 APPENDIX 1 UNITED STATES GOVERNMENT randum TO :AG, Mr. Edward F. Tennant DATE: MAY 5 1971 & FROM : AA/AJ?R, Samuel C. Adams, Jr.& SUBJECT: GAO Draft Report llEffectiveness of Technical Service Contracts in Providing U.S. Technical Assistance to East Africa" Introduction Subject draft report concludes that ten technical service contracts reviewed in East Africa do not contain precise statements of objectives and goals nor criteria by which progress toward objectives and goals may be measured; the applicable AID project planning and programming documents are equally deficient in these respects. The report also notes briefly that despite the aforementioned deficiencies creditable progress was made under some of the contracts, e.g., successful introduction of Vocational Agriculture courses in Kenyan secondary schools and strengthening dairying and livestock ranching cooperatives in Uganda. Other points discussed were occasional slowness of contractors to staff teams; one host country (Kenya) was sometimes slow in issuing work permits; qualified participant candidates are not always available in desired numbers or on time; and host governments sometimes change priorities in the middle of a project. GAO recommended that AID - - Take appropriate action to require that planning and programming documents, and technical service contracts as well, contain specific delineation of objectives and criteria by which progress may be evaluated. .=. Institute procedures to ascertain on a continuing basis - The seriousness of contract staffing problems including identification of contractors who fail to provide staff on a timely basis. - The full significance of the non-availability for training of participant candidates. Z--I this memorandum we give primary attention to the subjects covered in the recommendations. Bay U.S. Savings BOA Rcg&dy on the Payroll Savings Phn 48 APPENDIX I The draft report places strong emphasis on the inadequacy of AID project planning and programmin g documentation with respect to state- ments of project objectives and establishment of bench marks by which implementation of projects may be measured. Contracts for technical services supplied by universities or other types of organizations suffer from the same inadequacies. AD agrees that this makes it difficult for GAO to evaluate the effectiveness of technical service contracts in East Africa. Having recognized this problem quite some time ago, AD has expended a sizable amount of effort in reorganizing, planning, and establishing documentation systems in order to correct deficiencies. In 1967, a new system was initiated for preparing life-of-project proposals (PROP), annual implementation plans (PIP), and project appraisal reviews (PAR). More emphasis was placed upon evaluation, specification of objectives and measurable progress indicators. Continuing modification of these interrelated systems has resulted in a new PAR format and till soon produce a new PROP format. It should be noted that GAO studied ten technical service contracts which were in effect before the above mentioned procedures were instituted, and that attempting to change on-going contracts over to new procedures is very difficult. Projects recently initiated and future projects till incorporate new, improved techniques and should produce substantial progress in this area0 The ten projects reviewed are typical in that they indicate the difficulty of clearly identifying the effect of AD-financed inputs and outputs. The primary reason is that AD inputs form only a small portion of the total effort often including inputs by other donors, which goes into implementation of a given project. As an example, AD financed six professionals out of a total of fifty working at Edgerton College in Kenya; therefore, it is impossible to accurately establish what portion of the college's overall improvement was due to efforts of the AD-financed team. Evaluation can assess total progress and can make a judgement that AD contributed to that progress. Occasionally, AD inputs make up the sole external means employed to achieve specific targets. Even in these instances, there may be problems of definition and evaluation because the process of achieving social change is never as neatly contrdazble as a laboratory experiment. 49 APPENDIX I In none of the ten cases surveyed in East Africa were AID inputs the sole means, Although additional efforts will result in improvement, it is not possible to assure completely satisfactory results in attributing causality in cases of joint ventures such as technical assistance. It is &suggested that the draft report recognize this fact by including on page 1, before the fifth complete paragraph, a new paragraph to the effect that "GAO recognized, in the course of its examination, that there are often real difficulties in stating anticipated results since AID inputs often form only a small part of the total input to a project; and results of this partial input are usually dependent upon the performance of elements of the other inputs. Nevertheless, whenever possible, precise statements of objectives and input/output relationships should be established.11 Although the discrepancies noted on Page 6 are valid, it should also be noted that AID has made a diligent effort to make contract and other programming documents more ,precise. It is suggested that an additional sentence be added to the top of page 7 to the effect that llAID's noteworthy efforts in improving project planning and prw amming documentation should be continued to successful completion and careful attention given to achieving similar improvement of technical service contracts." In addition it is suggested that a new paragraph be inserted on page 36 after the third paragraph to the effect that: YChis installation process has gone forward systematically since the autumn of 1970. Four regional training conferences of three days each were held for Evaluation Officers, Program Officers and Deputy Directors of U.S.A.I.D. Missions. These conferences have been followed by one-week training visits to each Mission by teams from Washington to work with Mission personnel in the actual evaluation of two projects per Mission. The evaluations have resulted in action decisions to improve implementation or to modi@ forward plans. Frequently, the evaluations have also been a vehicle for communication with the contractor and the host government which has clarified targets and responsibilities. In order to assist Missions in the conduct of rigorous evaluations, AID has issued worksheets and guidelines for project managers and an Evaluation Handbook for Evaluation Officers. Regular training in Washington is also being set up to help overcome the effects of personnel turnover.1f 50 APPENDIX I At the end of page 39 a new paragraph is suggested to the effect that: "AID also recognizes that keeping its personnel alert and effective requires continuing efforts by management at all levels, Thus the December 1968 circular mentioned above has been followed by several other circulars about ways to improve project manage- ment. The teams which visited FEssions to help install the new program evaluation system also conducted training sessions on other aspects of project management besides evaluation. A Handbook for Froject Managers has been issued and management training seminars are being given on a regular basis for field project managers. Project Management has been on the agenda of AID's regional Mission Director conferences. Despite such continuing efforts, however, it must be recognized that technical assistance personnel, either contract or direct-hire, will usually have to overcome local difficulties in the environment of the host country. The difficulties which these countries face in operating effectively, are among the most important reasons why they need technical assistance.11 The second two part recommendation on page 43 of the Report is accept- able. The ability of contractors to supply qualified staff in timely fashion is of crucial importance to many AID projects now. It will become even more so as increasing proportions of AID-financed projects are implemented by contractors. At present, difficulties in staffing are dealt with on a case-by-case; and AID officials involved with a given contract exert appropriate effort to urge the contractor to speedier and better performance in providing the required staff. AID also uses past performance in earlier contracts in selection of contractors for new jobs. , One difficulty which contractors cannot always surmount is that AID sometimes asks them to undertake jobs on short notice. This is especially difficult when professors and highly qualified educational administrators are to be recruited. A lead time of a least a year may be required in order to recruit well qualified people. Such people usually make arrangements for employment during the next academic year by March. If, therefore, a contracting university signs a contract with AID after March, it is sometimes impossible to recruit an individual to arrive before September of the following year. Moreover, unavoidable accidents occur to upset contractor efforts to place individuals at fixed dates. Contractors and AID must cantend with the fact that individuals and their families some- 51 APPENDIX I times change their minds about foreign service at the last moment after having initially expressed interest. It is doubtful that all serious lapses can be avoided. As for the problem of Government of Kenya delays in issuing work permits, contributing to additional delay in effective project staffing, it appears that the situation has improved as a result of USAID/Kenya's representation to the GOK. There have been no instances of delays cited since the GAO team visited Nairobi. It is suggested therefore that the Draft Report be changed as follows: On page 28 the last sentence should be deleted, lSee GAo note-1 On page 29 delete the second and third sentences; and substitute the following: [See GAO note.] "It would appear that these contacts have been effective. No delays have been reported since the visit of the GAO team in the spring of 1970.fr The second part of the recommendation calling for "...ascertaining on a continuing basis,... (b) the full significance of the non- availability for training of host country nationals . . . ..%s also acceptable. Over the years we feel that performance in this respect has improved and it can be expected to improve still further. Based on our experience, we can now plan project inputs, including partici- pant training, more realistically. In the past, there has been a tendency toward optimism and high hopes that participants would be found. We should not assume from this that there will be no further difficulty in this regard. AID does not and cannot control the pro- cess. As for the host governments, they face real difficulties in finding appropriate candidates for training; and when they find them, they must arrange for the candidates to be released from current employment to obtain the further training planned in a project. These problems are being sorted out as the LDCls experience and give increasingly close attention to priorities. Although the problem of shortages is yet to be resolved, LDC education system are turning out increasing numbers of graduates to fill skilled manpower needs. Decontrol upon issuance of the final GAO report. GAO note: The suggested changes to pages 28 and 29 have been made in the third and fourth paragraphs on page 27 of this report. 52 U.S GAO. Wash.,D.C. i . .. _. ..-:. Copies of this report are available from the U. S. General Accounting Off ice, Room 6417, 441 G Street, N W., Washington, D.C., 20548. Copies are provided without charge to Mem- bers of Congress, congressional committee staff members, Government officials, members of the press, college libraries, faculty mem- bers and students. The price to the general public is $1.00 a copy. Orders should be ac- companied by cash or check.
Criteria Needed for Measuring Technical Assistance Results and Contractor Performance
Published by the Government Accountability Office on 1971-10-20.
Below is a raw (and likely hideous) rendition of the original report. (PDF)