- -__--_--~_I___ IJnited States General Accounting Office GAO Report to the Chairman, Subcommittee on Commerce, Consumer, and Monetary Affairs, Committee on Government Operations, House of Representatives Apri I 1990 U.S. & FOREIGN COMMERCIAL SERVICE Management and Program Reviews Are of Questionable Value g MI lllllllllll . Ill 141421 GAO,‘NSIAD-W-82 united states General Accounting Office Wathhgton, D.C. 20548 National Security and International AfYibirs Division B-208993 April 20,lOOO The Honorable Doug Barnard, Jr. Chairman, Subcommittee on Commerce, Consumer, and Monetary Affairs Committee on Government Operations House of Representatives Dear Mr. Chairman: This report responds to your May 17, 1989, request that we analyze the program evaluation function of the U.S. & Foreign Commercial Service (uw), the export promotion arm of the Department of Commerce’s International Trade Administration. Reports issued by the program evaluation office are known as Management and Program Reviews. You expressedconcern that these reviews might duplicate audits and inspec- tions performed by Commerce’sInspector General (IG). Specifically, you asked us to (1) document the number and cost of these reviews and the IG inspections completed since 1984, (2) determine whether the reviews were performed in accordancewith generally acceptedgovernment auditing standards, (3) assessthe overall quality of these reviews, and (4) determine whether they should be continued, given the IG’S increased resourcesand statutory authority for reviewing us&Fcsoperations. Between January 1986 and September 1989, the u&m completed four Results in Brief evaluations at an average cost of approximately $20,700. Over the same period, the IG completed 26 inspections at an average cost of $12,726. The IG inspections and the US&FCS’ evaluations are not audits subject to generally acceptedgovernment auditing standards. The US&FCS considers its evaluations to be “management reviews,” as opposedto audits, and they are conducted in accordancewith us&m standards. The IG also completed four audits of US&KS operations over this period. The IG’S audits are required to be conducted in accordancewith generally accepted government auditing standards. The quality of the uS&Fc!s’ evaluations as a managementtool is dimin- ished by the technical deficiencies in evidence gathering and the devel- opment of findings, conclusions,and recommendations; it is also diminished by the narrow focus on administrative and budgetary issues. Moreover, no system has been developed to monitor the status of report recommendations. Consequently, the usefulness of the u-s’ reviews Page 1 GAO/NSlAD90-82 U.S. fk Foreign Ckmmerchl Service B2OSs93 , in evaluating export promotion programs, or in bringing about meaning- ful change,has been limited. Given the small number of reviews that have been made, the effort neededto improve the review process,and the fact that Commerce’sIG has been given increased responsibilities for conducting reviews of USB~FCSoperations, we question whether continuation of the present review processis the most effective use of available resources. The us&m, through its nationwide office network, carries out a wide Background range of services aimed at increasing the number of small and medium- sized firms making a successfulcommitment to foreign trade and encouraging existing exporters to expand into new markets. The us&m’ foreign operations consist of about 166 U.S. commercial officers and about 460 foreign service nationals at 123 posts in 66 countries. Domes- tically, USB~FCSexport promotion services are available from trade spe- cialists in 47 district offices and 22 branch offices. In 1989, um export promotion expenditures totaled about $80 million-$37 million for foreign operations and $14 million for domestic operations, with the remainder for Washington-basedactivities. In 1984, following a comprehensive review of the U~&FCS, the Commerce, Consumer, and Monetary Affairs Subcommittee of the House Committee on Government Operations recommendedthat the us@cscreate a pro- gram evaluation function. In responseto the Committee’s recommenda- tion, the u%@csinitiated a program to conduct periodic reviews of its domestic and overseasoperations. These evaluations, known as Manage- ment and Program Reviews (MPRS), were to assesscompliance with U&FCSpolicies and procedures, identify management weaknesses,and improve the efficiency and effectiveness of worldwide operations. Teams of International Trade Administration employees,under the guidance of the Director of US&F&Office of Planning and Management, carry out the evaluations. From January 1986 to September 1989, the US&FCS issued four MPR Comparison of reports-at a cost of approximately $20,700 per report-covering oper- Number and Cost of ations in six overseasposts: Ankara and Istanbul, Turkey; Vienna, Aus- MPRs and IG Reports tria; Bern, Switzerland; Dublin, Ireland; and Mexico City, Mexico. Reports for reviews of operations in Iraq, Egypt, and Algiers were in the draft stagesat the completion of our audit work in September 1989. Page 2 GAO/NSL4D-90-82 U.S. i%Foreign Commercial Service -. E2os903 During this sameperiod, the IG issued 26 inspections at an average cost of $12,726. In contrast to the U&WCS’reviews, most of the IG’S reviews were of domestic operations, which tend to be of shorter duration and have lower associatedtravel costs. More detailed cost information is presented in appendix I. The uwxx has not been able to meet the goals it established for per- forming the IJ~UWS reviews. In 1986, US&FCS established a goal of review- ing all U&WCS offices approximately once every 6 years. As part of this goal, the U&WCS planned to annually assessat least 10 of 47 domestic district offices and posts in 13 of the 66 countries with US&FCS representation. The Director of the US&Fcs’ Office of Planning and Management said that a lack of funds and previous director generals’ lack of commitment to these reviews causedfew to be completed. Other officials agreed with the Director’s comments but added that technical and procedural prob- lems have also plagued the program, such as the use of unsupported findings, the reports’ narrow focus on compliance issues,and the lack of follow-up to ensure that report recommendations are implemented. Commerce’sIG has been able to conduct about six inspections each year of US&FCS posts and district offices and a limited number of full audits of overseaspost activities. Prom January 1986 to September 1989, the IG conducted 19 inspections of UZ.%FQ district offices and 7 inspections at overseasposts. In addition, the IG completed four audits of U&WX opera- tions over the sameperiod at a cost of approximately $129,492. Section 2301 of the Omnibus Trade and Competitiveness Act of 1988 broadens the IG’S responsibilities for conducting reviews of US&FCS oper- ations. An IG official told us that the IG’S office now has additional resourcesto broaden its reviews of U&W%operations and plans are to assign three additional auditors to this area. Neither the us&m evaluations nor the IG’s inspections are required to Generally Accepteg be performed in accordancewith generally acceptedgovernment audit- Government Audihng ing standards, also known as “yellow book” standards. “Yellow book” Standards Not standards only pertain to audits of government organizations, programs, activities, and functions. The standards require, among other things, a Required II trained and qualified staff, an independent audit staff and audit unit, the exercise of due professional care in conducting the audit and prepar- ing related reports, and the presenceof quality controls. The us&m and Page 3 GAO/NSIAD9042 U.S. % Foreign Commercial Service the IG have established their own in-house evaluation standards, which require that evaluation results be documented and verified. IGinspections and us&~csevaluations are short-term assessmentsof compliance with stated policies and procedures designedto give mana- gers timely information about current operations. IG inspections and us&m reviews are generally similar in that they seek to identify existing and potential problem areas in agency operations. In doing so, both the us&~& review team and the IG’S inspection staff rely primarily on observations and interviews and reviews of documents and related materials. Audits, on the other hand, are comprehensive reviews of agency operations to determine whether management complies with rel- evant laws, regulations, and other pertinent procedures; they also deter- mine whether agency programs are managedefficiently and effectively. The President’s Council on Integrity and Efficiency, whose membership includes the statutory IGS, has taken the position that the inspection function is separate and distinct from audits and investigations and, as such, does not have to adhere to generally acceptedgovernment audit- ing standards. usgc~csmanagement said they consider its evaluations to be management reviews as opposedto audits and, therefore, not subject to “yellow book” standards. &cause the US&FCS program reviews are similar in purpose to IG inspections and are not audits, we do not take exception to the U&F& position that the in-house program evaluations do not have to adhere to “yellow book” standards. The evaluations are conducted in accordancewith the standards prescribed in the US&FCS’ operations manual. The quality and usefulness of the uswcs reviews are diminished by Technical Deficiencies technical deficiencies in evidence gathering and in the development of Affect Review Quality conclusions and recommendations-which raises doubts about the use- fulness of the evaluations as a managementtool. Cur examination of the four final UF&FCS reports and supporting documentation showed that the reviews were generally not conducted in accordancewith the USBEFCS’ own standards. The final reports contained many findings that were not supported by documentary evidence. In addition, in most cases,the final reports did not explicitly identify the underlying causesof reported problems. Page 4 GAO/NSWM2 U.S. & Foreign Csmmercial Service In responseto our draft report, US$FCS officials said that many of the technical problems we noted were due to “start-up” problems organiza- tions experience with new programs. Many Reported Findings Each of the four reports we reviewed contained a number of findings that were not adequately supported. Overall, more than half of the 123 Not Adequately Supported reported findings (63 percent) were not adequately documented. In reviewing the documentation and interview write-ups supporting the final reports’ findings and recommendations,we found that the support- ing work paper files were poorly organized, and we could not link many reported findings to a specific set of supporting documents. The files consisted of folders of loose papers and documents, sometimeswritten illegibly or in shorthand. The relevance of many documents in the work paper files was not apparent. US&IQstandards require that findings and recommendations be sup- ported by sufficient and relevant evidence,recorded in the form of work papers, organized in a coherent fashion, and retained at the conclusion of each review. Reporting unsupported findings and recommendations may lead managementto take unnecessaryor counterproductive correc- tive actions. One reason that unsupported findings and recommendations may have been reported is that the evaluation processdoesnot include a quality control mechanism. No one independent of the evaluation team reviews the accuracy, completeness,and relevance of data used to support report findings. Although most members of the evaluation team had previously served in budget or other administrative positions and appear to have had some management-analyst experience, team members are not required to meet any minimum training requirements before being assignedto the evalua- tion team. One senior International Trade Administration official expressedsurprise over the lack of formal training and preparation par- ticipants receive prior to their assignment to the evaluation team. The Director of the MPR program told us that team members undergo “nonformal” training prior to performing site evaluations. This training includes step-by-step discussionsof the MPR process.Given the condition of the work paper files and the other technical deficiencies we noted during our review, it is questionable whether this training is adequate for conducting complex, broad-basedreviews. Page5 GAO/NSIAJMO-S2 U.S.%ForeignCommerda.lService , B-B08988 Root Causeof Problems In the reports we reviewed, the root causesof problems with trade pro- Not Always Identified motion and support activities were not aIways clearly identified. As a result, someoverseaspost officials openly questioned the validity of reported findings and recommendations.In responseto the Senior Com- mercial Officer’s criticisms of the us&~@ draft report on Turkey, the final report was revised to acknowledge someof the resource problems constraining post operations. However, despite the acknowledgement of these resource constraints, the final report still listed recommendations to increase program activities and outreach efforts that would have required additional resources.In light of the post’s resource constraints and dim prospects for obtaining additional funds, we believe it would have been more meaningful to discussthe effectiveness of current pro- grams with an eye towards eliminating or de-emphasizingthe least effective ones. According to USBEFCX standards, each evaluation team member must pre- pare a work sheet for each finding identified. The work sheet is a device intended to assist team members in logically developing a finding and in documenting exceptional performance that may have broader applica- tions. The work sheet also helps the team member identify the basic ele- ments of a finding. One of the elements, cause,is a key factor in developing a finding and recommending corrective action. None of the work sheetswe reviewed identified the causeof the reported finding. The post officials we interviewed were generally critical of the MPR pro- cess.Several complained that the findings sometimesignored resource constraints or problems stemming from headquarters’ inadequaciesin managing program activities. For example, a us&~cs program evaluation report on operations at the Mexico City Trade Center dismissed claims that headquarters’ actions contributed to the post’s problems. However, a February 1990 Inspector General report on Trade Center activities supported the post’s position that management actions at US&FCS head- quarters were a primary causeof problems found in the Mexico City Trade Center. Several post officials expressedconcern that the US&FCS’ review team may be reluctant to identify UWIKBor other International Trade Admin- istration units as the causeof post problems. If an evaluation team mem- ber attributes the underlying causeof a post’s problem to arduous administrative policies or budgetary constraints, the team member would, in most cases,be essentially identifying his home unit as the causeof the problem. Page 0 GAO/NSIAD9O-S2 U.S. & Foreign Commercial Service Report Recommendations Neither the program evaluation staff nor other us&x8 officials were Not Followed Up able to tell us whether any of the posts reviewed had implemented MPR recommendations.We independently verified that 26 of the 67 recom- mendations made from the combined Austria and Switzerland review had not been implemented by the post. We did not independently deter- mine the status of the remaining recommendations. us&&xxstandards require that the status of recommendations be moni- tored to ensure that they are carried out within specified time frames. According to UEMCSstandards, staff from the US&FC$ Office of Foreign Operations, with the assistanceof staff from the us&rcs’ budget office, are responsible for ensuring that report recommendations are implemented. The MPR team is not tracking its recommendations. No follow-up mecha- nism has been established to ensure that recommendations are carried out. The Director of the MPR team confirmed that little or no effort has been made to ensure that the different US&FCS offices are carrying out report recommendations. Reviews Focus on The MPRS completed to date have not focused on managementconcerns Administrative and Budget that transcend the activities of individual posts. Rather, the review efforts have focused, to a large extent, on the post’s compliance with Issues administrative and budgetary procedures. The MPR processwas devel- oped to give management the information neededto make informed management,program, and resource allocation decisions.However, 76 (62 percent) of the 123 findings we analyzed involved administrative or budgetary issues,such as whether administrative forms were completed correctly or whether cash collections and other funds were being promptly submitted to headquarters, as specified in the us@~Y opera- tions manual. According to a senior International Trade Administration official, the lack of a clear agreement among us&rcsofficials as to what the MPRS are supposedto accomplish has resulted in the MPR processbecoming a com- pliance function that deals with noncontentious issues-as opposed to management reviews focusing on issuesthat concern program effective- ness and resource utilization. While we recognizethat periodic assessmentsof compliance with admin- istrative procedures are necessary,we question the value of a manage- ment review processthat doesnot attempt to assessthe effectiveness of particular export promotion programs, the efficiency with which these programs are carried out, or the contributions that particular posts make to overall agency objectives. The reviews were not broad enough in scopeto allow for the assessmentof program results acrossseveral posts and geographical regions or the analysis of the overall effective- nessof export promotion programs within a particular post. The scope and objective of recent reviews were not shaped by management con- cerns about particular export promotion issuesor the effectiveness of a particular program. As part of any reorientation towards broad-basedmanagement reviews, the evaluations would have to cover a sufficient number of posts to allow for a systematic and widespread assessmentof a particular export promotion program. Moreover, team members would have to develop program evaluation skills as well as possessa good understanding of the US&W program and organizational objectives. In responseto questions we posed about the effectiveness of the present program, senior UWZFCS officials told us they are considering undertaking steps to refocus and broaden the scopeof MPRS to help ensure that these reviews address managementpriorities and issuesof concern. As part of this reorientation, staff from the USFCSOffice of Foreign Operations, who monitor and supervise the day-to-day activities of overseasposts, would be assignedto the evaluation teams to help ensure that the teams focus on management issues. Internal review functions are established as a managementprerogative Conclusion and can be a valuable managementtool. At someadditional cost, which we did not attempt to quantify, the technical deficiencies we identified can be corrected. Formal training could be provided to team members, an internal control mechanism could be established to improve report quality, recommendation follow-up could be strengthened, and a consen- sus as to what these reviews are intended to achieve could be reached by senior USCFCS management. However, given the limited number of MPRS that have been made, the effort neededto improve the MPR process,and the fact that Commerce’s IG has been granted increased responsibilities for conducting reviews of wwcs operations, we question whether continuation of the present review processis the most effective use of available resources. Pages In commenting on the findings in our draft report, us&m officials told us of a number of steps that would be taken to improve the MPR process. Although these are steps in the right direction, it is still questionable whether they will correct the fundamental deficiencies we found in the program. We recommend that the Secretary of Commercedirect the Under Secre- Recormnendation tary of the International Trade Administration to abolish the us&~& current Management and Program Review program and use these resourcesto enhance other ongoing management and monitoring activi- ties of the us&rcs.If the US&F& management insists on maintaining the program, then the deficiencies cited in this report need to be corrected to improve the quality of the review process. As requested, we did not obtain formal agency comments on this report; however, we discussedit with the appropriate CommerceDepartment officials and have made modifications to reflect their comments, as appropriate. As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date it is issued. At that time, we will send copies to the Secretary of Commerceand appropriate congressionalcommittees and make copies available to other interested parties upon request. The report was prepared under the direction of Allan I. Mendelowitz, Director, Trade, Energy, and Finance Issues,(202) 275-4812.The princi- pal GAO staff members responsible for this review were Benjamin Nel- son, Assistant Director, Stephen Lord, Evaluator-in-Charge, and Robert Shields, Evaluator. Sincerely yours, Frank C. Conahan Assistant Comptroller General Page 9 Appendix I i Number and Cost of MPR Reviews Completed ‘+ Table 1.1:Co8t of Program Evaluatlons Made by the US&FCS’ Program Evaluation Unit (1985-1989) Post 1985 1986 1987 1988 1989 Total Completed -- revlewo Boston, Mass8 $11,491 $934 $0 $0 $0 $12,425 -Lisbon, Portugala 2,967 9,313 0 0 0 12,280 --Vienna, Austria, and Bern, Switzerland 0 26,165 5,196 0 0 31,361 Dublin, Ireland 0 0 0 0 19,853 19,853 --- Ankara and Istanbul, Turkey 0 0 0 0 19,853 19,853 -__Mexico City, Mexico 0 0 0 0 11,593 11,593 Review8 In procecrr as of September 30,1989 Ewpt 0 0 0 0 21,510 21,510 Algeria __--~ ~..... 0 0 0 0 21,510 21,510 Iraq .-_-- 0 0 0 0 4,964 4,964 Field teats oi forms and other procedure8 Norfolk, Virginia 0 0 0 0 3,409 3,409 New -. York, New-_-.. - .._.. York 0 0 0 0 3,885 3,885 Total $14,458 $35,412 $5,196 $0 $108,577 $162,643 Note:The abovetable representsunauditedfigures that maynot be directly comparableto table 1.2. 8Pilotstudies.Finalreportsneverissued. Source:US&FCS. Page 10 GAO/NXAD9O-S2 U.S. fk Foreign Commercial Service Number and Cast of MPR Itevtews completea Table 1.2:CortofIGIn~pectlon~( 1985101989) Subject 1985 1988 1887 1988 1989 TOtal Europe Export Development Office $27,189 $0 $0 $0 $0 $27,189 Houston D.0 16,407 0 0 0 0 18,407 San Diego D.O. 4,630 0 0 0 0 4,830 San Francisco D.O. 19,016 0 0 0 0 19,016 Phoenix DO. 0 12,707 0 0 0 12.707 Oklahoma City D.0 0 11,243 0 0 0 11,243 Salt Lake City-D.O. 0 9,111 0 0 0 9,111 Hartford D.O. 0 15,061 0 0 0 15,081 San Juan D.O. 0 12,452 0 0 0 12.452 Mexico City 0 0 14,565 0 0 14;585 Canadian posts 0 0 22,122 0 0 22,122 Mexico Citv 0 0 17.210 0 0 17.210 San Diego D.O. 0 0 10,078 0 0 10,078 Phoenixb.0. 0 0 6,200 0 0 8;200 New York D.O. 0 0 16,182 0 0 18,182 Des Moines D.O. 0 0 10,612 0 0 10.812 Des Moines D.O. 0 0 0 10,612 0 101812 Salt Lake City D.O. 0 0 0 2,266 0 2,288 Chicaao D.O. 0 0 0 0 15.634 15,834 Trinidad & Tobago 0 0 0 0 11,682 11,882 Baltimore D.O. 0 0 0 0 10,553 10,553 Barbados 0 0 0 0 11,682 11,882 Dominican Reoublic 0 0 0 0 11,682 11.882 Mexico City 0 0 0 0 10,458 10,458 Mexico -~--- City 0 0 0 0 7,084 7,084 Indianapolis D.O. --.----- 0 0 0 0 14,441 14,441 Total $87,242 $80,574 $98,989 $12.878 $93.218 $330.879 Note: The above table represents unaudited figures that may not be directly comparable to table 1.1. Source: Department of Commerce IG. Page 11 GAO/NSW~2 U.S. 8 Foreign Chnmercial Service Appendix II Objectives,Scope,and Methodology At the request of the Chairman of the Subcommittee on Commerce,Con- sumer, and Monetary Affairs, House Committee on Government Opera- tions, we reviewed the U.S. & Foreign Commercial Service’sprogram evaluation efforts. The objectives of our review were to (1) document the number and cost of the us&m’ Management and Program Reviews and the IG’S inspections completed since 1984, (2) determine whether these reviews were performed in accordancewith generally accepted government auditing standards, (3) evaluate the overall quality of these reviews, and (4) assesswhether they should be continued, given the IG’s increased resourcesand statutory authority for reviewing USE~ZFCS operations. We reviewed all final evaluation reports and supporting documents for every evaluation made between January 1986 and September 1989. To obtain a better understanding of the us&r& foreign post operations and to gaugethe extent to which report recommendations had been imple- mented at overseassites, we visited US&FCS posts in Vienna, Austria, and Bern, Switzerland. We also interviewed various US&FCS officials, includ- ing senior commercial officers, foreign service nationals at the overseas posts we visited, USLFCS headquarters officials, and staff from Com- merce’s Office of Inspector General. We assessedthe relative cost and productivity of the MPR team and Com- merce’s IG team using cost data provided by each office. We did not inde- pendently verify the accuracy of the data. We performed our review between May 1989 and September 1989 in accordancewith generally acceptedgovernment auditing standards. (483622) Page 12 GAO/NSIAD-90432 U.S. & Foreign Commercial !3ervice
U.S. & Foreign Commercial Service: Management and Program Reviews Are of Questionable Value
Published by the Government Accountability Office on 1990-04-20.
Below is a raw (and likely hideous) rendition of the original report. (PDF)