U.S. & Foreign Commercial Service: Management and Program Reviews Are of Questionable Value

Published by the Government Accountability Office on 1990-04-20.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

-   -__--_--~_I___   IJnited   States   General   Accounting   Office

GAO                  Report to the Chairman, Subcommittee
                     on Commerce, Consumer, and Monetary
                     Affairs, Committee on Government
                     Operations, House of Representatives

Apri I 1990
                     U.S. & FOREIGN
                     COMMERCIAL SERVICE
                     Management and
                     Program Reviews Are
                     of Questionable Value


                   united states
                   General Accounting Office
                   Wathhgton, D.C. 20548

                   National Security and
                   International AfYibirs Division

                   April 20,lOOO
                   The Honorable Doug Barnard, Jr.
                   Chairman, Subcommittee on Commerce,
                     Consumer, and Monetary Affairs
                   Committee on Government Operations
                   House of Representatives
                   Dear Mr. Chairman:

                   This report responds to your May 17, 1989, request that we analyze the
                   program evaluation function of the U.S. & Foreign Commercial Service
                   (uw),     the export promotion arm of the Department of Commerce’s
                   International Trade Administration. Reports issued by the program
                   evaluation office are known as Management and Program Reviews. You
                   expressedconcern that these reviews might duplicate audits and inspec-
                   tions performed by Commerce’sInspector General (IG). Specifically, you
                   asked us to (1) document the number and cost of these reviews and the
                   IG inspections completed since 1984, (2) determine whether the reviews
                   were performed in accordancewith generally acceptedgovernment
                   auditing standards, (3) assessthe overall quality of these reviews, and
                   (4) determine whether they should be continued, given the IG’S increased
                   resourcesand statutory authority for reviewing us&Fcsoperations.

                   Between January 1986 and September 1989, the u&m completed four
Results in Brief   evaluations at an average cost of approximately $20,700. Over the same
                   period, the IG completed 26 inspections at an average cost of $12,726.
                   The IG inspections and the US&FCS’ evaluations are not audits subject to
                   generally acceptedgovernment auditing standards. The US&FCS     considers
                   its evaluations to be “management reviews,” as opposedto audits, and
                   they are conducted in accordancewith us&m standards. The IG also
                   completed four audits of US&KS operations over this period. The IG’S
                   audits are required to be conducted in accordancewith generally
                   accepted government auditing standards.
                   The quality of the uS&Fc!s’ evaluations as a managementtool is dimin-
                   ished by the technical deficiencies in evidence gathering and the devel-
                   opment of findings, conclusions,and recommendations; it is also
                   diminished by the narrow focus on administrative and budgetary issues.
                   Moreover, no system has been developed to monitor the status of report
                   recommendations. Consequently, the usefulness of the u-s’       reviews

                   Page 1                            GAO/NSlAD90-82   U.S. fk Foreign Ckmmerchl Service
                      B2OSs93                                                                    ,

                      in evaluating export promotion programs, or in bringing about meaning-
                      ful change,has been limited.
                      Given the small number of reviews that have been made, the effort
                      neededto improve the review process,and the fact that Commerce’sIG
                      has been given increased responsibilities for conducting reviews of
                      USB~FCSoperations, we question whether continuation of the present
                      review processis the most effective use of available resources.

                      The us&m, through its nationwide office network, carries out a wide
Background            range of services aimed at increasing the number of small and medium-
                      sized firms making a successfulcommitment to foreign trade and
                      encouraging existing exporters to expand into new markets. The us&m’
                      foreign operations consist of about 166 U.S. commercial officers and
                      about 460 foreign service nationals at 123 posts in 66 countries. Domes-
                      tically, USB~FCSexport promotion services are available from trade spe-
                      cialists in 47 district offices and 22 branch offices. In 1989, um
                      export promotion expenditures totaled about $80 million-$37 million
                      for foreign operations and $14 million for domestic operations, with the
                      remainder for Washington-basedactivities.

                      In 1984, following a comprehensive review of the U~&FCS, the Commerce,
                      Consumer, and Monetary Affairs Subcommittee of the House Committee
                      on Government Operations recommendedthat the us@cscreate a pro-
                      gram evaluation function. In responseto the Committee’s recommenda-
                      tion, the u%@csinitiated a program to conduct periodic reviews of its
                      domestic and overseasoperations. These evaluations, known as Manage-
                      ment and Program Reviews (MPRS), were to assesscompliance with
                      U&FCSpolicies and procedures, identify management weaknesses,and
                      improve the efficiency and effectiveness of worldwide operations.
                      Teams of International Trade Administration employees,under the
                      guidance of the Director of US&F&Office of Planning and Management,
                      carry out the evaluations.

                      From January 1986 to September 1989, the US&FCS     issued four MPR
Comparison of         reports-at a cost of approximately $20,700 per report-covering oper-
Number and Cost of    ations in six overseasposts: Ankara and Istanbul, Turkey; Vienna, Aus-
MPRs and IG Reports   tria; Bern, Switzerland; Dublin, Ireland; and Mexico City, Mexico.
                      Reports for reviews of operations in Iraq, Egypt, and Algiers were in the
                      draft stagesat the completion of our audit work in September 1989.

                      Page 2                         GAO/NSL4D-90-82 U.S. i%Foreign Commercial Service

                     During this sameperiod, the IG issued 26 inspections at an average cost
                     of $12,726. In contrast to the U&WCS’reviews, most of the IG’S reviews
                     were of domestic operations, which tend to be of shorter duration and
                     have lower associatedtravel costs. More detailed cost information is
                     presented in appendix I.
                     The uwxx has not been able to meet the goals it established for per-
                     forming the IJ~UWS   reviews. In 1986, US&FCS established a goal of review-
                     ing all U&WCS offices approximately once every 6 years. As part of this
                     goal, the U&WCS planned to annually assessat least 10 of 47 domestic
                     district offices and posts in 13 of the 66 countries with US&FCS
                     The Director of the US&Fcs’ Office of Planning and Management said that
                     a lack of funds and previous director generals’ lack of commitment to
                     these reviews causedfew to be completed. Other officials agreed with
                     the Director’s comments but added that technical and procedural prob-
                     lems have also plagued the program, such as the use of unsupported
                     findings, the reports’ narrow focus on compliance issues,and the lack of
                     follow-up to ensure that report recommendations are implemented.

                     Commerce’sIG has been able to conduct about six inspections each year
                     of US&FCS posts and district offices and a limited number of full audits of
                     overseaspost activities. Prom January 1986 to September 1989, the IG
                     conducted 19 inspections of UZ.%FQ   district offices and 7 inspections at
                     overseasposts. In addition, the IG completed four audits of U&WX opera-
                     tions over the sameperiod at a cost of approximately $129,492.

                     Section 2301 of the Omnibus Trade and Competitiveness Act of 1988
                     broadens the IG’S responsibilities for conducting reviews of US&FCS    oper-
                     ations. An IG official told us that the IG’S office now has additional
                     resourcesto broaden its reviews of U&W%operations and plans are to
                     assign three additional auditors to this area.

                     Neither the us&m evaluations nor the IG’s inspections are required to
Generally Accepteg   be performed in accordancewith generally acceptedgovernment audit-
Government Audihng   ing standards, also known as “yellow book” standards. “Yellow book”
Standards Not        standards only pertain to audits of government organizations, programs,
                     activities, and functions. The standards require, among other things, a
Required     II      trained and qualified staff, an independent audit staff and audit unit,
                     the exercise of due professional care in conducting the audit and prepar-
                     ing related reports, and the presenceof quality controls. The us&m and

                     Page 3                           GAO/NSIAD9042   U.S. % Foreign Commercial Service
                         the IG have established their own in-house evaluation standards, which
                         require that evaluation results be documented and verified.
                         IGinspections and us&~csevaluations are short-term assessmentsof
                         compliance with stated policies and procedures designedto give mana-
                         gers timely information about current operations. IG inspections and
                         us&m reviews are generally similar in that they seek to identify
                         existing and potential problem areas in agency operations. In doing so,
                         both the us&~& review team and the IG’S inspection staff rely primarily
                         on observations and interviews and reviews of documents and related
                         materials. Audits, on the other hand, are comprehensive reviews of
                         agency operations to determine whether management complies with rel-
                         evant laws, regulations, and other pertinent procedures; they also deter-
                         mine whether agency programs are managedefficiently and effectively.
                         The President’s Council on Integrity and Efficiency, whose membership
                         includes the statutory IGS, has taken the position that the inspection
                         function is separate and distinct from audits and investigations and, as
                         such, does not have to adhere to generally acceptedgovernment audit-
                         ing standards.
                         usgc~csmanagement said they consider its evaluations to be management
                         reviews as opposedto audits and, therefore, not subject to “yellow
                         book” standards. &cause the US&FCS program reviews are similar in
                         purpose to IG inspections and are not audits, we do not take exception to
                         the U&F& position that the in-house program evaluations do not have
                         to adhere to “yellow book” standards. The evaluations are conducted in
                         accordancewith the standards prescribed in the US&FCS’    operations

                         The quality and usefulness of the uswcs reviews are diminished by
Technical Deficiencies   technical deficiencies in evidence gathering and in the development of
Affect Review Quality    conclusions and recommendations-which raises doubts about the use-
                         fulness of the evaluations as a managementtool. Cur examination of the
                         four final UF&FCS reports and supporting documentation showed that the
                         reviews were generally not conducted in accordancewith the USBEFCS’
                         own standards. The final reports contained many findings that were not
                         supported by documentary evidence. In addition, in most cases,the final
                         reports did not explicitly identify the underlying causesof reported

                         Page 4                         GAO/NSWM2      U.S. & Foreign Csmmercial Service
                                In responseto our draft report, US$FCS officials said that many of the
                                technical problems we noted were due to “start-up” problems organiza-
                                tions experience with new programs.

Many Reported Findings   Each of the four reports we reviewed contained a number of findings
                         that were not adequately supported. Overall, more than half of the 123
Not Adequately Supported reported findings (63 percent) were not adequately documented.
                                In reviewing the documentation and interview write-ups supporting the
                                final reports’ findings and recommendations,we found that the support-
                                ing work paper files were poorly organized, and we could not link many
                                reported findings to a specific set of supporting documents. The files
                                consisted of folders of loose papers and documents, sometimeswritten
                                illegibly or in shorthand. The relevance of many documents in the work
                                paper files was not apparent.
                                US&IQstandards require that findings and recommendations be sup-
                                ported by sufficient and relevant evidence,recorded in the form of work
                                papers, organized in a coherent fashion, and retained at the conclusion
                                of each review. Reporting unsupported findings and recommendations
                                may lead managementto take unnecessaryor counterproductive correc-
                                tive actions.

                                One reason that unsupported findings and recommendations may have
                                been reported is that the evaluation processdoesnot include a quality
                                control mechanism. No one independent of the evaluation team reviews
                                the accuracy, completeness,and relevance of data used to support
                                report findings.
                                Although most members of the evaluation team had previously served
                                in budget or other administrative positions and appear to have had some
                                management-analyst experience, team members are not required to meet
                                any minimum training requirements before being assignedto the evalua-
                                tion team. One senior International Trade Administration official
                                expressedsurprise over the lack of formal training and preparation par-
                                ticipants receive prior to their assignment to the evaluation team. The
                                Director of the MPR program told us that team members undergo
                                “nonformal” training prior to performing site evaluations. This training
                                includes step-by-step discussionsof the MPR process.Given the condition
                                of the work paper files and the other technical deficiencies we noted
                                during our review, it is questionable whether this training is adequate
                                for conducting complex, broad-basedreviews.

                                Page5                          GAO/NSIAJMO-S2   U.S.%ForeignCommerda.lService

Root Causeof Problems   In the reports we reviewed, the root causesof problems with trade pro-
Not Always Identified   motion and support activities were not aIways clearly identified. As a
                        result, someoverseaspost officials openly questioned the validity of
                        reported findings and recommendations.In responseto the Senior Com-
                        mercial Officer’s criticisms of the us&~@ draft report on Turkey, the
                        final report was revised to acknowledge someof the resource problems
                        constraining post operations. However, despite the acknowledgement of
                        these resource constraints, the final report still listed recommendations
                        to increase program activities and outreach efforts that would have
                        required additional resources.In light of the post’s resource constraints
                        and dim prospects for obtaining additional funds, we believe it would
                        have been more meaningful to discussthe effectiveness of current pro-
                        grams with an eye towards eliminating or de-emphasizingthe least
                        effective ones.
                        According to USBEFCX standards, each evaluation team member must pre-
                        pare a work sheet for each finding identified. The work sheet is a device
                        intended to assist team members in logically developing a finding and in
                        documenting exceptional performance that may have broader applica-
                        tions. The work sheet also helps the team member identify the basic ele-
                        ments of a finding. One of the elements, cause,is a key factor in
                        developing a finding and recommending corrective action. None of the
                        work sheetswe reviewed identified the causeof the reported finding.
                        The post officials we interviewed were generally critical of the MPR pro-
                        cess.Several complained that the findings sometimesignored resource
                        constraints or problems stemming from headquarters’ inadequaciesin
                        managing program activities. For example, a us&~cs program evaluation
                        report on operations at the Mexico City Trade Center dismissed claims
                        that headquarters’ actions contributed to the post’s problems. However,
                        a February 1990 Inspector General report on Trade Center activities
                        supported the post’s position that management actions at US&FCS    head-
                        quarters were a primary causeof problems found in the Mexico City
                        Trade Center.
                        Several post officials expressedconcern that the US&FCS’  review team
                        may be reluctant to identify UWIKBor other International Trade Admin-
                        istration units as the causeof post problems. If an evaluation team mem-
                        ber attributes the underlying causeof a post’s problem to arduous
                        administrative policies or budgetary constraints, the team member
                        would, in most cases,be essentially identifying his home unit as the
                        causeof the problem.

                        Page 0                          GAO/NSIAD9O-S2 U.S. & Foreign Commercial Service
Report Recommendations      Neither the program evaluation staff nor other us&x8 officials were
Not Followed Up             able to tell us whether any of the posts reviewed had implemented MPR
                            recommendations.We independently verified that 26 of the 67 recom-
                            mendations made from the combined Austria and Switzerland review
                            had not been implemented by the post. We did not independently deter-
                            mine the status of the remaining recommendations.
                            us&&xxstandards require that the status of recommendations be moni-
                            tored to ensure that they are carried out within specified time frames.
                            According to UEMCSstandards, staff from the US&FC$     Office of Foreign
                            Operations, with the assistanceof staff from the us&rcs’ budget office,
                            are responsible for ensuring that report recommendations are
                            The MPR team is not tracking its recommendations. No follow-up mecha-
                            nism has been established to ensure that recommendations are carried
                            out. The Director of the MPR team confirmed that little or no effort has
                            been made to ensure that the different US&FCS offices are carrying out
                            report recommendations.

Reviews Focus on            The MPRS completed to date have not focused on managementconcerns
Administrative and Budget   that transcend the activities of individual posts. Rather, the review
                            efforts have focused, to a large extent, on the post’s compliance with
Issues                      administrative and budgetary procedures. The MPR processwas devel-
                            oped to give management the information neededto make informed
                            management,program, and resource allocation decisions.However, 76
                            (62 percent) of the 123 findings we analyzed involved administrative or
                            budgetary issues,such as whether administrative forms were completed
                            correctly or whether cash collections and other funds were being
                            promptly submitted to headquarters, as specified in the us@~Y opera-
                            tions manual.
                            According to a senior International Trade Administration official, the
                            lack of a clear agreement among us&rcsofficials as to what the MPRS are
                            supposedto accomplish has resulted in the MPR processbecoming a com-
                            pliance function that deals with noncontentious issues-as opposed to
                            management reviews focusing on issuesthat concern program effective-
                            ness and resource utilization.
                            While we recognizethat periodic assessmentsof compliance with admin-
                            istrative procedures are necessary,we question the value of a manage-
                            ment review processthat doesnot attempt to assessthe effectiveness of
             particular export promotion programs, the efficiency with which these
             programs are carried out, or the contributions that particular posts
             make to overall agency objectives. The reviews were not broad enough
             in scopeto allow for the assessmentof program results acrossseveral
             posts and geographical regions or the analysis of the overall effective-
             nessof export promotion programs within a particular post. The scope
             and objective of recent reviews were not shaped by management con-
             cerns about particular export promotion issuesor the effectiveness of a
             particular program.
             As part of any reorientation towards broad-basedmanagement reviews,
             the evaluations would have to cover a sufficient number of posts to
             allow for a systematic and widespread assessmentof a particular export
             promotion program. Moreover, team members would have to develop
             program evaluation skills as well as possessa good understanding of the
             US&W    program and organizational objectives.
             In responseto questions we posed about the effectiveness of the present
             program, senior UWZFCS   officials told us they are considering undertaking
             steps to refocus and broaden the scopeof MPRS to help ensure that these
             reviews address managementpriorities and issuesof concern. As part of
             this reorientation, staff from the USFCSOffice of Foreign Operations,
             who monitor and supervise the day-to-day activities of overseasposts,
             would be assignedto the evaluation teams to help ensure that the teams
             focus on management issues.

             Internal review functions are established as a managementprerogative
Conclusion   and can be a valuable managementtool. At someadditional cost, which
             we did not attempt to quantify, the technical deficiencies we identified
             can be corrected. Formal training could be provided to team members,
             an internal control mechanism could be established to improve report
             quality, recommendation follow-up could be strengthened, and a consen-
             sus as to what these reviews are intended to achieve could be reached
             by senior USCFCS management.
             However, given the limited number of MPRS that have been made, the
             effort neededto improve the MPR process,and the fact that Commerce’s
             IG has been granted increased responsibilities for conducting reviews of
             wwcs operations, we question whether continuation of the present
             review processis the most effective use of available resources.

                  In commenting on the findings in our draft report, us&m officials told
                  us of a number of steps that would be taken to improve the MPR process.
                  Although these are steps in the right direction, it is still questionable
                  whether they will correct the fundamental deficiencies we found in the

                  We recommend that the Secretary of Commercedirect the Under Secre-
Recormnendation   tary of the International Trade Administration to abolish the us&~&
                  current Management and Program Review program and use these
                  resourcesto enhance other ongoing management and monitoring activi-
                  ties of the us&rcs.If the US&F& management insists on maintaining the
                  program, then the deficiencies cited in this report need to be corrected to
                  improve the quality of the review process.
                  As requested, we did not obtain formal agency comments on this report;
                  however, we discussedit with the appropriate CommerceDepartment
                  officials and have made modifications to reflect their comments, as
                  As arranged with your office, unless you publicly announce its contents
                  earlier, we plan no further distribution of this report until 30 days from
                  the date it is issued. At that time, we will send copies to the Secretary of
                  Commerceand appropriate congressionalcommittees and make copies
                  available to other interested parties upon request.
                  The report was prepared under the direction of Allan I. Mendelowitz,
                  Director, Trade, Energy, and Finance Issues,(202) 275-4812.The princi-
                  pal GAO staff members responsible for this review were Benjamin Nel-
                  son, Assistant Director, Stephen Lord, Evaluator-in-Charge, and Robert
                  Shields, Evaluator.
                  Sincerely yours,

                  Frank C. Conahan
                  Assistant Comptroller General

                  Page 9
  Appendix I

i Number and Cost of MPR Reviews Completed ‘+

  Table 1.1:Co8t of Program Evaluatlons Made by the US&FCS’ Program Evaluation Unit (1985-1989)
  Post                                             1985        1986          1987          1988                       1989               Total
  --            revlewo
     Boston, Mass8                               $11,491       $934             $0           $0                          $0           $12,425
 -Lisbon, Portugala                                2,967       9,313             0             0                          0            12,280
 --Vienna, Austria, and Bern, Switzerland              0      26,165         5,196             0                          0            31,361
     Dublin, Ireland                                   0           0             0             0                     19,853            19,853
 --- Ankara and Istanbul, Turkey                       0           0             0             0                     19,853            19,853
  -__Mexico City, Mexico                               0           0             0             0                     11,593            11,593

  Review8 In procecrr as of September 30,1989
    Ewpt                                                 0                0               0               0          21,510            21,510
 __--~         ~.....                                    0                0               0               0          21,510            21,510
 .-_--                                                   0                0               0               0           4,964             4,964

  Field teats oi forms and other procedure8
     Norfolk, Virginia                                  0               0                0                0         3,409               3,409
       -. York, New-_-..
 - .._..               York                             0               0                0                0         3,885               3,885
 Total                                            $14,458         $35,412           $5,196               $0      $108,577            $162,643
                                              Note:The abovetable representsunauditedfigures that maynot be directly comparableto table 1.2.

                                              Page 10                                   GAO/NXAD9O-S2 U.S. fk Foreign Commercial Service
                                          Number and Cast of MPR Itevtews completea

Table 1.2:CortofIGIn~pectlon~(   1985101989)
Subject                                         1985             1988             1887              1988             1989               TOtal
Europe Export Development Office              $27,189               $0               $0                $0               $0            $27,189
Houston D.0                                    16,407                0                0                 0                0             18,407
San Diego D.O.                                  4,630                0                0                 0                0              4,830
San Francisco D.O.                             19,016                0                0                 0                0             19,016
Phoenix DO.                                         0           12,707                0                 0                0             12.707
Oklahoma City D.0                                   0           11,243                0                 0                0             11,243
Salt Lake City-D.O.                                 0            9,111                0                 0                0              9,111
Hartford D.O.                                       0           15,061                0                 0                0             15,081
San Juan D.O.                                       0           12,452                0                 0                0             12.452
Mexico City                                         0                0           14,565                 0                0             14;585
Canadian posts                                      0                0          22,122                  0                0             22,122
Mexico Citv                                         0                0           17.210                 0                0             17.210
San Diego D.O.                                      0                0           10,078                 0                0             10,078
Phoenixb.0.                                         0                0            6,200                 0                0              8;200
New York D.O.                                       0                0           16,182                 0                0             18,182
Des Moines D.O.                                     0                0           10,612                 0                0             10.812
Des Moines D.O.                                     0                0                0            10,612                0             101812
Salt Lake City D.O.                                 0                0                0             2,266                0              2,288
Chicaao D.O.                                        0                0                0                 0           15.634             15,834
Trinidad & Tobago                                   0                0                0                 0           11,682             11,882
Baltimore D.O.                                      0                0                0                 0           10,553             10,553
Barbados                                            0                0                0                 0           11,682             11,882
Dominican Reoublic                                  0                0                0                 0           11,682             11.882
Mexico City                                         0                0                0                 0           10,458             10,458
-~---   City                                        0                0                0                 0            7,084              7,084
Indianapolis   D.O.
         --.-----                                   0                0                0                 0           14,441             14,441
Total                                        $87,242          $80,574          $98,989           $12.878          $93.218            $330.879
                                         Note: The above table represents unaudited figures that may not be directly comparable to table 1.1.

                                         Source: Department of Commerce IG.

                                         Page 11                                      GAO/NSW~2           U.S. 8 Foreign Chnmercial Service
Appendix II

Objectives,Scope,and Methodology

              At the request of the Chairman of the Subcommittee on Commerce,Con-
              sumer, and Monetary Affairs, House Committee on Government Opera-
              tions, we reviewed the U.S. & Foreign Commercial Service’sprogram
              evaluation efforts. The objectives of our review were to (1) document
              the number and cost of the us&m’ Management and Program Reviews
              and the IG’S inspections completed since 1984, (2) determine whether
              these reviews were performed in accordancewith generally accepted
              government auditing standards, (3) evaluate the overall quality of these
              reviews, and (4) assesswhether they should be continued, given the IG’s
              increased resourcesand statutory authority for reviewing USE~ZFCS
              We reviewed all final evaluation reports and supporting documents for
              every evaluation made between January 1986 and September 1989. To
              obtain a better understanding of the us&r& foreign post operations and
              to gaugethe extent to which report recommendations had been imple-
              mented at overseassites, we visited US&FCS  posts in Vienna, Austria, and
              Bern, Switzerland. We also interviewed various US&FCS officials, includ-
              ing senior commercial officers, foreign service nationals at the overseas
              posts we visited, USLFCS headquarters officials, and staff from Com-
              merce’s Office of Inspector General.

              We assessedthe relative cost and productivity of the MPR team and Com-
              merce’s IG team using cost data provided by each office. We did not inde-
              pendently verify the accuracy of the data.
              We performed our review between May 1989 and September 1989 in
              accordancewith generally acceptedgovernment auditing standards.

(483622)      Page 12                        GAO/NSIAD-90432 U.S. & Foreign Commercial !3ervice