oversight

Millennium Challenge Corporation: Results of Transportation Infrastructure Projects in Seven Countries

Published by the Government Accountability Office on 2012-09-12.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                 United States Government Accountability Office

GAO              Report to Congressional Committees




September 2012
                 MILLENNIUM
                 CHALLENGE
                 CORPORATION
                 Results of
                 Transportation
                 Infrastructure
                 Projects in Seven
                 Countries




GAO-12-631
                                              September 2012

                                              MILLENNIUM CHALLENGE CORPORATION
                                              Results of Transportation Infrastructure Projects in
                                              Seven Countries
Highlights of GAO-12-631, a report to
congressional committees




Why GAO Did This Study                        What GAO Found
To help developing countries reduce           The Millennium Challenge Corporation (MCC)—a U.S. government corporation—
poverty and stimulate economic                recognizes the importance of a disciplined, transparent, and accountable
growth, MCC has approved 26 bilateral         approach to tracking compact results. However, it reduced the scopes of its early
compact agreements totaling about             transportation infrastructure projects and reports mixed success in meeting key
$9.3 billion. In the seven compacts that      performance targets. In addition, problems with data quality call into question the
ended in 2010 and 2011—Honduras,              reliability of those reported results. GAO found the following for the seven
Cape Verde, Nicaragua, Vanuatu,               compacts ending in 2010 and 2011, each with a road project or a port project.
Georgia, Armenia, and Benin—
transportation infrastructure projects        Road Projects
generally received about 50 percent of
the compact’s total funding. To               • MCC reduced kilometers to be paved under six compacts—Honduras, Cape
measure the results of its compacts,            Verde, Nicaragua, Vanuatu, Georgia, and Armenia—by a combined 63
MCC sets targets for various                    percent (from about 1,800 to 600 km) because of increased construction costs
performance indicators—such as                  and political problems in partner governments. MCC reported meeting
number of kilometers paved or volume            reduced targets for five compacts. However, for three compacts, MCC did not
of merchandise passing through a                consistently account for kilometers completed with funding from third parties.
port—and estimates the number of
beneficiaries. This report, responding
                                              • MCC reported meeting revised targets for road roughness—a measure of
to a congressional mandate, examines            pavement quality—for five of the compacts. However, reported data have
the extent to which MCC has, for                quality issues, including the inconsistent application of measurement
transportation infrastructure projects,         methodologies and calculation errors that resulted in overstated results.
(1) achieved expected performance             • MCC reported meeting targets for annual average daily traffic—a measure of
targets and (2) consistently estimated          the volume of traffic using the road—for three of the compacts. However,
numbers of beneficiaries. GAO
                                                weaknesses in traffic baseline estimates may have affected the establishment
analyzed MCC documents, interviewed
                                                of targets and therefore MCC’s ability to measure results.
MCC officials, and drew on fieldwork
completed for related work in four of         Port Projects
the seven countries.
                                              • In Cape Verde, MCC funding ($53.7 million) was insufficient to construct all
                                                planned port elements. As a result, MCC reduced the project’s scope and
What GAO Recommends
                                                deferred measuring the results of key indicators.
MCC should strengthen existing
policies and practices regarding              • In Benin, MCC completed most of the envisioned scope of the port project.
measuring and evaluating results data           MCC’s data show that the compact met the original target for one of three key
and formalize a quality review process          performance indicators, volume of merchandise. (The other two indicators
to improve its beneficiary calculations.        relate to the measurement of shipping costs.) However, GAO found that
MCC agreed with all of our                      MCC’s estimation of this indicator’s baseline may lead to overstated results. In
recommendations and outlined some               addition, data quality reviews identified problems with the data used, which
steps the agency will take or has               MCA-Benin did not formally address.
already taken to address them.
                                              In 2009, MCC improved its methodology for estimating beneficiaries—people
                                              who realize income gains or expenditure savings as a result of its investment—
                                              by standardizing its approach. MCC subsequently revised its beneficiary
                                              numbers for all compacts. However, the new approach did not include a formal
                                              quality review process. As a result, implementation of the new approach suffered
                                              from varying degrees of quality problems. For example, (1) MCC did not
                                              implement its beneficiary estimation methodology consistently across early
View GAO-12-631. For more information,        transportation infrastructure projects; (2) beneficiary calculations contained
contact David Gootnick at (202) 512-3149 or   incorrect formulas and numbers, and differed from supporting documents; and
gootnickd@gao.gov.                            (3) beneficiary figures in MCC’s public documents were sometimes inaccurate.
                                                                                      United States Government Accountability Office
Contents


Letter                                                                                                1
                       Background                                                                     4
                       MCC Data Show that Some Key Performance Targets Were Met,
                         but Data Quality Problems Call into Question Reported Results              11
                       Data Resulting from Beneficiary Calculations May Not Be Valid                23
                       Conclusions                                                                  27
                       Recommendations for Executive Action                                         28
                       Agency Comments and Our Evaluation                                           29

Appendix I             Objectives, Scope, and Methodology                                           31



Appendix II            Compact Timeframes                                                           34



Appendix III           Roads Data                                                                   36



Appendix IV            Port Data                                                                    39



Appendix V             Comments from the Millennium Challenge Corporation                           40



Appendix VI            GAO Contacts and Staff Acknowledgments                                       43



Related GAO Products                                                                                44



Tables
                       Table 1: Key Performance Results for the Kilometers of Road
                                Completed Indicator, MCC Compacts Ending in 2010
                                and 2011                                                            12
                       Table 2: Scope Revisions of Road Projects for MCC Compacts
                                Ending in 2010 and 2011                                             13




                       Page i                               GAO-12-631 MCC Transportation Infrastructure
          Table 3: Key Performance Results for the Road Roughness
                   Indicator, Compacts Ending in 2010 and 2011                        14
          Table 4: Key Performance Results for the Average Annual Daily
                   Traffic Indicator, Compacts Ending in 2010 and 2011                16
          Table 5: Data Quality Reviews Performed for Road Project
                   Common Indicators                                                  18
          Table 6: Key Performance Results for MCC Port Project in
                   Cape Verde                                                         20
          Table 7: Key Performance Results for MCC Port Project in Benin              21
          Table 8: Kilometers of Roads Completed for MCC Road Projects in
                   Compacts Ending in 2010 and 2011                                   36
          Table 9: Road Roughness of MCC Road Projects in Compacts
                   ending in 2010 and 2011, as Measured by the International
                   Roughness Index (IRI)                                              36
          Table 10: Average Annual Daily Traffic on MCC-Funded Roads in
                   Compacts Ending in 2010 and 2011                                   37
          Table 11: Volume of Merchandise through Port for Projects in MCC
                   Compacts That Ended in 2010 and 2011                               39
          Table 12: Container Ship Time at Berth for Port Projects in MCC
                   Compacts That Ended in 2010 and 2011                               39
          Table 13: Container Ship Time at Anchor for Port Projects in MCC
                   Compacts That Ended in 2010 and 2011                               39


Figures
          Figure 1: Funds for Transportation Infrastructure and Other
                   Projects, by MCC Compact                                             6
          Figure 2: Relationship between MCC Indicators, Baselines, Targets,
                   and Results                                                          8
          Figure 3: Timeframes for MCC Compacts Ending in 2010 and 2011,
                   by Date of Signature                                               35




          Page ii                             GAO-12-631 MCC Transportation Infrastructure
Abbreviations
IRI        International Roughness Index
MCC        Millennium Challenge Corporation
MCA        Millennium Challenge Account

This is a work of the U.S. government and is not subject to copyright protection in the
United States. The published product may be reproduced and distributed in its entirety
without further permission from GAO. However, because this work may contain
copyrighted images or other material, permission from the copyright holder may be
necessary if you wish to reproduce this material separately.




Page iii                                     GAO-12-631 MCC Transportation Infrastructure
United States Government Accountability Office
Washington, DC 20548




                                   September 12, 2012

                                   The Honorable Patrick Leahy
                                   Chairman
                                   The Honorable Lindsey Graham
                                   Ranking Member
                                   Subcommittee on the Department of State,
                                     Foreign Operations, and Related Programs
                                   Committee on Appropriations
                                   United States Senate

                                   The Honorable Kay Granger
                                   Chairwoman
                                   The Honorable Nita Lowey
                                   Ranking Member
                                   Subcommittee on State, Foreign Operations, and
                                     Related Programs
                                   Committee on Appropriations
                                   House of Representatives

                                   The Millennium Challenge Corporation (MCC), a U.S. government
                                   corporation, was established in 2004 to provide aid to developing
                                   countries that have demonstrated a commitment to ruling justly,
                                   encouraging economic freedom, and investing in people. MCC provides
                                   assistance to eligible countries through multiyear compact agreements to
                                   fund specific projects aimed at reducing poverty and stimulating economic
                                   growth. MCC’s core principles include assessing the results of its funding
                                   to determine how its activities affect poverty and economic growth. To do
                                   this, MCC has instituted a monitoring and evaluation approach that
                                   includes identifying relevant indicators to measure results during and at
                                   the end of project implementation and establishing performance targets
                                   for each indicator. MCC’s efforts to determine how its activities will affect
                                   poverty and economic growth also include estimating the number of
                                   people who will benefit from its projects and analyzing the impact of its
                                   projects on populations of specific interest.




                                   Page 1                                GAO-12-631 MCC Transportation Infrastructure
As of June 2012, MCC had signed 26 compacts, committing a total of
approximately $9.3 billion. 1 Seven of these compacts ended in 2010 or
2011: Honduras, Cape Verde, Nicaragua, Vanuatu, Georgia, Armenia,
and Benin. 2 (See app. II for compact time frames.)

In the fiscal year 2008 Consolidated Appropriations Act, Congress
mandated that we review the results achieved by MCC compacts. 3 We
have previously reported on the results of MCC’s transportation
infrastructure projects in four countries. In July 2011, we reported that
MCC had met reduced targets for its compacts with Cape Verde and
Honduras, the first two compacts to reach completion. 4 In June 2012, we
reported that MCC transportation infrastructure projects in Georgia and
Benin varied in quality and may not be sustainable. 5 Because much of
MCC’s funding for compacts that ended in 2010 and 2011 focused on
transportation infrastructure projects, this report examines the extent to
which MCC has, for transportation infrastructure projects, (1) achieved
expected performance targets and (2) used a consistent methodology to
estimate numbers of beneficiaries.




1
  MCC commits funding when a compact is signed and obligates funds after the compact
enters into force. As of June 2012, MCC had signed initial compacts with, in order of
signature, Madagascar, Honduras, Cape Verde, Nicaragua, Georgia, Benin, Vanuatu,
Armenia, Ghana, Mali, El Salvador, Mozambique, Lesotho, Morocco, Mongolia, Tanzania,
Burkina Faso, Namibia, Senegal, Moldova, the Philippines, Jordan, Malawi, Indonesia,
and Zambia. In February 2012, MCC signed a second compact with Cape Verde.
2
  We did not include the Madagascar compact in this review because, as the result of an
undemocratic transfer of power in Madagascar in March 2009, MCC formally terminated
the compact effective August 31, 2009, before the end of the 5-year statutory compact
timeframe.
3
 Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, § 668(d)(1)(A). The Act also
required us to examine the financial control and procurement practices of MCC and its
accountable entities. We responded to this requirement separately in GAO, Millennium
Challenge Corporation: MCC Has Addressed a Number of Implementation Challenges,
but Needs to Improve Financial Controls and Infrastructure Planning, GAO-10-52
(Washington, D.C.: Nov. 6, 2009).
4
 GAO, Millennium Challenge Corporation: Compacts in Cape Verde and Honduras
Achieved Reduced Targets, GAO-11-728 (Washington, D.C.: Jul. 25, 2011).
5
 GAO, Millennium Challenge Corporation: Georgia and Benin Transportation
Infrastructure Projects Varied in Quality and May Not Be Sustainable, GAO-12-630
(Washington, D.C.: Jun. 27, 2012).




Page 2                                      GAO-12-631 MCC Transportation Infrastructure
To assess the extent to which MCC has achieved its performance targets
for transportation infrastructure projects under compacts that ended in
2010 and 2011, we reviewed MCC guidance and policy documents and
analyzed compact agreements, monitoring and evaluation plans, and
results data. We compared actual results achieved at the end of the
compact for select performance indicators with MCC’s original targets
and, in some cases, with revised targets associated with each indicator.
Since MCC tracks many performance indicators for each compact project,
we selected a subset of indicators to examine for this report.

•   For the road projects, we examined kilometers of road completed,
    roughness as a measure of road quality, and average annual daily
    traffic. We selected those road project indicators because they
    address the projects’ key objectives and because they are among the
    “common indicators” MCC requires for all road projects, so that it can
    aggregate results across countries.

•   For the port projects, we examined the volume of merchandise
    through the port and two indicators related to the measurement of
    shipping costs: container ship time at berth and container ship time at
    anchor. Because MCC has not established common indicators for
    ports, we selected indicators that addressed the projects’ key
    objectives, corresponded to the common road indicators, and were
    originally included in both port projects.

Additionally, we interviewed MCC officials about data quality and indicator
tracking tables. As part of our related engagements—focused on Cape
Verde and Honduras, and on Georgia and Benin 6—we visited compact
projects in-country and met with partner country government officials,
including those responsible for compact implementation, and with
contractors, project managers, construction supervisors, and relevant
private businesses.

To assess the extent to which MCC has consistently applied its
beneficiary estimation methodology, we reviewed MCC’s Guidelines for
Economic and Beneficiary Analysis as well as its beneficiary calculations
and accompanying supporting documents for the seven compacts that
ended in 2010 and 2011. We compared original and revised beneficiary
estimates for the transportation projects and assessed the justification


6
GAO-11-728 and GAO-12-630.




Page 3                               GAO-12-631 MCC Transportation Infrastructure
                   behind any changes. Data are considered reliable when they are
                   accurate, complete, consistent, and valid, given the uses for which they
                   are intended.

                   MCC enters into a legal relationship with partner country governments
                   that vest an accountable entity with responsibility for day-to-day
                   management of compact project implementation, including monitoring
                   and evaluation activities such as setting and revising performance
                   targets. Because such actions require MCC’s direct oversight and
                   approval, throughout this report, we attribute all decisions related to
                   project rescoping and compact targets to MCC. (See app. I for further
                   details of our objectives, scope, and methodology.)

                   We conducted this performance audit from August 2011 to September
                   2012, as part of a body of work on this subject, in accordance with
                   generally accepted government auditing standards. Those standards
                   require that we plan and perform the audit to obtain sufficient, appropriate
                   evidence to provide a reasonable basis for our findings and conclusions
                   based on our audit objectives. We believe the evidence obtained provides
                   a reasonable basis for our findings and conclusions based on our audit
                   objectives.



Background

MCC Organization   MCC is managed by a chief executive officer, appointed by the President
                   with the advice and consent of the Senate, and is overseen by a Board of
                   Directors. The Secretary of State serves as board chair and the Secretary
                   of the Treasury as vice-chair. 7 MCC’s model of providing foreign aid is
                   based on a set of core principles deemed essential for effective
                   development assistance, including good governance, country ownership,
                   focus on results, and transparency. According to MCC, country ownership
                   of an MCC compact occurs when a country’s national government
                   controls the prioritization process during compact development, is
                   responsible for implementation, and is accountable to its domestic


                   7
                    Other board members are the U.S. Trade Representative, the Administrator of the U.S.
                   Agency for International Development, the Chief Executive Officer of MCC, and up to four
                   Senate-confirmed nongovernmental members appointed by the President from lists of
                   individuals submitted by congressional leadership.




                   Page 4                                      GAO-12-631 MCC Transportation Infrastructure
                          stakeholders for decision making and results. To implement the compact,
                          the partner government establishes an accountable entity, referred to as
                          a Millennium Challenge Account (MCA). 8 MCC provides the framework
                          and guidance for compact implementation, monitoring, and evaluation
                          that MCAs are to use in implementing compact projects.


Transportation            The seven MCC compacts completed in 2010 and 2011 each included a
Infrastructure Projects   transportation infrastructure project that—except in the case of Armenia—
                          received 50 percent or more of the compact’s total funding (see fig.1). For
                          the purposes of this report, we have defined transportation infrastructure
                          as public works that convey passengers or goods from one place to
                          another; infrastructure includes structures such as roads, seaports,
                          airports, and railways. We reviewed five compacts with road projects, one
                          with a port project, and one with both types of projects.




                          8
                           MCC generally refers to each country’s accountable entity by combining MCA with the
                          country’s name (e.g., MCA-Benin).




                          Page 5                                     GAO-12-631 MCC Transportation Infrastructure
Figure 1: Funds for Transportation Infrastructure and Other Projects, by MCC Compact




                                        Notes: Amount at signature is the amount allocated when MCC and the partner country sign the
                                        compact; amount disbursed is the actual amount disbursed by compact closure. Any undisbursed
                                        funds are returned to MCC. “Other projects” includes funds disbursed before the compact’s entry into
                                        force, to facilitate implementation of the compact.




                                        Page 6                                            GAO-12-631 MCC Transportation Infrastructure
MCC Monitoring and     According to MCC’s Policy for Monitoring and Evaluation of Compacts
Evaluation Framework   and Threshold Programs, 9 performance monitoring helps track progress
                       toward compact goals and objectives and serves as a management tool.
                       Therefore, MCC requires partner countries to create a monitoring and
                       evaluation plan that they update throughout the course of the compact.
                       The plan’s monitoring component outlines the performance indicators by
                       which the compact results will be measured. It also establishes a
                       performance target for each indicator, including a baseline value (the
                       starting value of an indicator) and the expected year by which the target
                       will be achieved. 10 For some indicators, baseline measurements are
                       necessary to establish interim and final targets. MCC and the MCAs
                       monitor the progress of compact activities using an indicator tracking
                       table, which is a reporting tool that displays targets and tracks progress
                       against them. (See fig. 2.)




                       9
                        MCC developed guidelines in 2006 to assist eligible countries in the preparation of
                       monitoring and evaluation plans and issued an updated policy in 2009 and in 2012.
                       10
                         In many cases, indicators are chosen because they relate to inputs for the economic
                       rate of return analysis that MCC conducts to estimate a project’s likely impact on the
                       partner country’s economic growth and poverty reduction prior to compact approval. More
                       specifically, a project’s estimated economic rate of return is the expected annual average
                       return to the country’s firms, individuals, or sectors for each dollar that MCC spends on the
                       project.




                       Page 7                                        GAO-12-631 MCC Transportation Infrastructure
Figure 2: Relationship between MCC Indicators, Baselines, Targets, and Results




                                        Notes: The indicators shown are for a road project. The terms indicator, baseline values, actual
                                        results, performance targets, end-of-compact target, and data quality reviews pertain to all MCC
                                        projects.




                                        Page 8                                            GAO-12-631 MCC Transportation Infrastructure
Transportation Infrastructure   MCC established common indicators in 2009 to aggregate results across
Indicators                      countries. 11 For road projects, MCC has established several common
                                indicators, including the following: 12

                                •    Kilometers of road completed: A measure of the total length of road
                                     completed under a compact. This indicator is a cumulative measure of
                                     works completed.

                                •    Road roughness: A measure of pavement quality that affects ride
                                     quality, vehicle delay costs, fuel consumption, and maintenance costs.
                                     Roughness is used to determine the reduction in road-user costs and
                                     indicates when resurfacing work is needed. New pavement roughness
                                     ratings—as measured by the international roughness index (IRI)—
                                     should range from 1.5 to 3.5; the higher the IRI, the rougher the
                                     pavement, with a roughness of 0 being perfectly smooth.

                                •    Average annual daily traffic: A measure of the volume and type of
                                     traffic using a road.

                                MCC has not established common indicators for ports, because ports are
                                a small part of its portfolio (it has funded port projects in only two
                                countries). However, we selected certain key indicators to review
                                because they addressed the projects’ key objective of economic growth: 13




                                11
                                  MCC issued its Common Indicators Directive in 2009, establishing the common
                                indicators and requiring MCAs to use them in project monitoring and evaluation plans.
                                Although the seven compacts that we reviewed were signed before 2009, initial monitoring
                                and evaluation plans for three of the six compacts with road projects included common
                                indicators for kilometers completed, road roughness, and average annual daily traffic. The
                                MCAs for the three remaining compacts introduced the common indicators into their
                                monitoring and evaluation activities before the compacts ended.
                                12
                                  MCC has developed additional common indicators for roads, including the percentage of
                                funds disbursed for contracted studies, value of signed contracts for road works,
                                percentage of funds for contracted road works disbursed, kilometers of roads under works
                                contracts, and value of signed contracts for feasibility design supervision and program
                                management contracts.
                                13
                                  MCC has developed other indicators for ports, including percentage of funds for
                                contracted port works disbursed, average time for goods to clear customs, average
                                duration of stay of trucks at port, tons of merchandise shipped per year, and whether the
                                port is meeting the International Ship and Port Facility Code, which establishes security
                                requirements for ports.




                                Page 9                                       GAO-12-631 MCC Transportation Infrastructure
                           •   Volume of merchandise through the port: A measure of the total
                               volume of exports and imports passing through the port each year.

                           •   Container ship time at berth: A measure of the average number of
                               days a container ship spends at berth to load and/or unload its cargo.

                           •   Container ship time at anchor: A measure of the average number of
                               hours a container ship spends at anchor waiting for a berth.

Independent Data Quality   MCC requires that MCAs contract an independent entity (such as a local
Reviews                    or international firm, research organization, or individual consultant) to
                           review the quality of compact performance data. These reviews
                           determine the utility, objectivity, and integrity of information gathered,
                           identify problems with data quality, and recommend actions to remedy
                           these issues. Reviews should ensure that data used to measure
                           indicators meet the following criteria: validity, reliability, timeliness,
                           precision, and integrity. MCC policy allows each country to determine the
                           frequency and timing of each review as well as the indicators to be
                           evaluated. This requirement was fulfilled for every compact except
                           Nicaragua. MCC policy requires the MCA to comment on the review,
                           including noting the recommendations it will implement. In addition, the
                           MCA should reflect, in a subsequent monitoring and evaluation plan, any
                           indicator changes made in response to data quality reviews. MCC policy
                           also requires the MCA to make publically available on its website a
                           summary of the final reviews and its comments. MCAs are responsible for
                           ensuring that MCC-approved recommendations are implemented.


Beneficiary Assessment     MCC defines beneficiaries as people who realize income gains or
                           expenditure savings as a result of its investment. 14 MCC’s authorizing
                           legislation requires that compacts contain an identification of the intended
                           beneficiaries, disaggregated by income level, gender, and age, to the



                           14
                             According to its Guidelines for Economic and Beneficiary Analysis, MCC takes the
                           household as the unit of measurement, counting as beneficiaries all members of the
                           household. MCC’s projects include targeted projects that benefit specific individuals or
                           households—for example, projects focused on agricultural development or school
                           construction—but also includes projects that are national or broad-based in scope and
                           benefit people living in a large geographical area. Beneficiaries include everyone who
                           realizes income gain or expenditure savings regardless of the magnitude of these
                           changes, although the income gain or expenditure savings are likely to vary significantly
                           across individuals and projects.




                           Page 10                                      GAO-12-631 MCC Transportation Infrastructure
                          maximum extent practicable. MCC notes that “the use of these
                          [beneficiary] analyses and their availability to the general public are
                          trademarks of MCC’s commitment to transparency and results-based
                          aid.” In addition, MCC’s 2009 Guidelines for Economic and Beneficiary
                          Analysis maintain that identification of intended beneficiaries should not
                          be limited to counting the number of people who benefit from compacts
                          but should include a more detailed analysis of the effect of compacts on
                          different subgroups, including the poor, the elderly, women, children, and
                          ethnic subpopulations. MCC has not undertaken such detailed beneficiary
                          analysis for the seven compacts completed in 2010 or 2011. 15


                          MCC reduced the scope of early road and port projects and reports mixed
MCC Data Show that        success in meeting key performance targets. In addition, problems with
Some Key                  data quality call those reported results into question.
Performance Targets
Were Met, but Data
Quality Problems Call
into Question
Reported Results

MCC Reduced Targets for   In each of the six compacts with road projects, MCC reduced the number
Completed Roads by 63     of kilometers to be completed. In addition, the roughness and annual
Percent and Had Data      average daily traffic indicators had data quality problems. Most compacts
                          did not examine the three common road indicators in their data quality
Quality Problems          reviews. In addition, MCC did not consistently account for third-party
                          funding.




                          15
                            MCC officials stated that detailed beneficiary analyses for its compacts are carried out
                          retrospectively. MCC provided us with detailed beneficiary analyses, including “poverty
                          scorecards” focusing on the impact of compacts on the poor, for six other compacts out of
                          the 26 current and completed compacts.




                          Page 11                                      GAO-12-631 MCC Transportation Infrastructure
According to MCC Data, Most    MCC reduced by 63 percent the total number of road kilometers to be
Compacts Met Reduced Targets   completed for the six compacts with road projects, from an original
for Kilometers of Roads        combined target of 1,822.8 kilometers to a final combined target of 623.9
Completed                      kilometers. 16 Five of six compacts met their reduced targets for kilometers
                               completed. Honduras was the only compact that did not meet its target
                               (see table 1).

                               Table 1: Key Performance Results for the Kilometers of Road Completed Indicator,
                               MCC Compacts Ending in 2010 and 2011

                                                            Original          Revised     Percentage            Final
                                   Country                    target            target        change           result     Target met
                                   Hondurasa                       200.0        174.5              -12.8        115.0           x
                                   Cape Verde                       63.0         39.3              -37.6         40.6           ○
                                   Nicaragua                       158.0         67.0              -57.6         74.0           ○
                                   Georgia                         245.0        220.2              -10.1       220.2b           ○
                                   Vanuatu                         213.8        149.7              -30.0        149.7           ○
                                   Armenia                         943.0         24.4              -97.4         24.4           ○
                                   Total                      1,822.8           675.1              -63.0        623.9

                               Legend:
                               ●    Met original target
                               ○    Met revised target
                               x    Did not meet original or revised target
                               Source: GAO analysis of MCC data.

                               Note: See appendix III for additional details on targets and results for the kilometers of road indicator
                               in each country.
                               a
                                Kilometers include secondary roads and sections 3 and 4 of the highway in Honduras. Sections 1
                               and 2 of the highway were to be completed post-compact with additional funding provided by the
                               Central American Bank for Economic Integration; as of March 9, 2012, construction of these
                               segments was not complete.
                               b
                                Of this amount, 217 kilometers were fully rehabilitated using MCC funds. For the remaining about 3
                               kilometers, MCC funded maintenance items, such as road painting.


                               In general, MCC decreased the scope of the road projects because of
                               increased construction costs, changes in exchange rates, or actions
                               taken by host country governments that were inconsistent with MCC’s


                               16
                                 In response to a draft copy of this report, MCC noted that if one excludes road projects
                               that MCC terminated or discontinued as a result of undemocratic behavior by the
                               governments of Armenia and Nicaragua, the final target for kilometers of roads completed
                               in Honduras, Georgia, Cape Verde, and Vanuatu is only 19 percent below the original
                               target. See appendix V for a copy of MCC’s comments.




                               Page 12                                              GAO-12-631 MCC Transportation Infrastructure
                                            eligibility criteria. For example, MCC decreased the scope of the
                                            compacts in Nicaragua and Armenia because of concerns about
                                            democratic governance in the respective partner governments
                                            (see table 2).

Table 2: Scope Revisions of Road Projects for MCC Compacts Ending in 2010 and 2011

Compact      Scope change                                                            Reason for scope change
Honduras     Reduced kilometers of secondary roads completed.                        Increased construction costs.
             Removed two sections of the highway project, to be
             completed with additional funding from the Central
             American Bank for Economic Integration post-compact.
Cape Verde   Eliminated two roads.                                                   Increased construction costs.
Nicaragua    In June 2009, MCC terminated funding for the                            Electoral irregularities in the municipal elections of
             transportation activities not already under contract. At this           November 2008, which were inconsistent with MCC
             time, 74 kilometers of road were already under                          eligibility criteria.a
             construction.
Georgia      Reduced kilometers of road to 70 percent of the original                Increased construction costs and changes in exchange
             plan.                                                                   rates.
             Subsequently increased kilometers of road to roughly 90                 MCC obligated an additional $100 million to the compact
             percent of the original plan.                                           following the end of Georgia’s war with Russia over South
                                                                                     Ossetia in 2008, $60 million of which went to the road
                                                                                     project.
Vanuatu      Reduced scope from the rehabilitation of several roads,                 Increased construction costs and changes in exchange
             airstrips and wharfs to two roads.                                      rates.
             Increased kilometers completed on road in Santo.                        New Zealand provided additional funding to support MCC
                                                                                     road project in June 2009.
Armenia      Reduced kilometers of road to roughly 32 percent of the                 Increased construction costs and changes in exchange
             original plan prior to June 2008.                                       rates.
             In June 2009, MCC put an indefinite hold on funding for                 Concerns about the status of democratic governance in
             road construction that was not already underway. At this                Armenia, which were inconsistent with MCC’s eligibility
                                                                                               a
             time, construction of 24.4 kilometers had been completed.               criteria.
                                            Source: GAO analysis of MCC documents.
                                            a
                                             The country also failed to meet MCC’s corruption indicator, which measures the extent to which
                                            public power is exercised for private gain.




                                            Page 13                                               GAO-12-631 MCC Transportation Infrastructure
MCC-Reported Results for   MCC data show that five of six compacts met original targets for road
Road Roughness Have Data   roughness. In addition, MCC reports that the Georgia and Armenia
Quality Issues             compacts surpassed their original roughness targets and built a smoother
                           road than originally planned. 17 Honduras did not did not meet its
                           roughness target. (See table 3.)

                           Table 3: Key Performance Results for the Road Roughness Indicator, Compacts
                           Ending in 2010 and 2011

                               Country                  Original target    Revised target         Final result      Target met
                               Hondurasa                            2.2                                     3.0          x
                               Cape Verde                           2.3                                     2.3          ●
                               Nicaragua                            3.1                                     2.5          ●
                               Georgia                              3.2                   2.5               1.5         ●○
                               Vanuatu                              3.5                                     3.0          ●
                               Armenia                              5.0                   4.0               3.5         ●○
                           Legend:
                           ●    Met original target
                           ○    Met revised target
                           x    Did not meet original or revised target
                           Source: GAO analysis of MCC data.

                           Notes: A higher roughness number indicates a rougher road, while zero is a perfectly smooth road. A
                           missing value for the revised target indicates that the original target was not revised. See appendix III
                           for additional details on targets and results for the roughness indicator in each country.
                           a
                            The roughness measure includes secondary roads and sections 3 and 4 of the highway in Honduras.
                           Sections 1 and 2 of the highway were to be completed post-compact with additional funding provided
                           by the Central American Bank for Economic Integration; as of March 9, 2012, construction of these
                           segments was not complete.


                           We identified problems with the measurement of road roughness for
                           some of the six compacts with road projects, including the use of


                           17
                              In Georgia, MCC reduced the roughness target to account for a delayed construction
                           start date. A road’s surface is smoothest directly following construction. Because the delay
                           in construction pushed the expected completion date closer to the compact end date, this
                           reduced the amount of time MCC-funded roads could deteriorate before roughness
                           measurements were taken at the end of the compact. Therefore, the roads were expected
                           to have a better roughness measure than originally planned. In Armenia, MCC reduced
                           the target for road roughness once the project was placed on indefinite hold. The 24.4
                           kilometers that MCC completed before putting the project on hold had a lower roughness
                           target than the full package of roads MCC originally planned to complete. Therefore,
                           removing the terminated roads from the project resulted in a better roughness measure for
                           the compact overall.




                           Page 14                                             GAO-12-631 MCC Transportation Infrastructure
inconsistent application of methodologies and calculation errors. These
problems call into question MCC’s reported roughness results.

Inconsistent application of methodologies. MCC determined
roughness using a visual scale for some roads and mechanical
measurements for others. While both methods are legitimate means of
measuring roughness, they have different degrees of accuracy.
Moreover, MCC did not apply either measurement method consistently
across compacts. First, for the two countries measuring roughness
visually, Cape Verde and Vanuatu, MCC inappropriately provided the
MCAs a World Bank scale for unpaved roads to assess the paved roads.
Second, for compacts where roughness was measured mechanically, the
MCAs used different machines that had varying levels of accuracy. Third,
for three compacts—Nicaragua, Honduras, and Armenia—the MCAs
measured roughness on both sides of the road and then averaged these
measurements to calculate a final measure. In contrast, MCA-Georgia
calculated roughness using measurements taken on one side of the road
only, potentially producing a less accurate measurement since roughness
can vary from one side of the road to the other. 18

Calculation errors. We found that errors calculating roughness led to an
overstatement of results. For example, MCA-Nicaragua recorded zeros—
an IRI rating for a perfectly smooth road—for sections that could not be
measured, incorrectly lowering the overall roughness calculation from 4.8
to 3.4 for one road and indicating that the road was smoother than it
actually was. 19 Additionally, MCA-Honduras did not account for the
differing lengths of each segment of secondary road when aggregating
the roughness values and therefore overstated its reported results at 3.2
instead of 3.3. Although this difference is not large, consistent
methodologies should be used across compacts to allow a valid basis for
comparing and evaluating results.




18
  During construction, because all lanes of a road often are not paved at the same time,
uneven construction quality (e.g., nonuniform compaction of asphalt pavements),
construction techniques that allow some variation in the surface, and other factors may
lead to different roughness measures for each lane.
19
  For two of the road segments, the nonmeasurable sections of road were left blank and
the calculation of the final IRI used the value from the one measurable side of the road as
the overall measure for that kilometer.




Page 15                                      GAO-12-631 MCC Transportation Infrastructure
MCC-Reported Results for         MCC data show that three of the six compacts with road projects met
Road Traffic Have Data Quality   original or revised targets for average annual daily traffic. 20 The
Issues                           Nicaragua and Armenia compacts met their original targets and the
                                 Vanuatu compact met a reduced target. For the other three compacts,
                                 Cape Verde did not meet its original target; Georgia did not meet its
                                 reduced target; and MCC stated that a final traffic measure in Honduras is
                                 not expected until late 2012. (See table 4.)

                                 Table 4: Key Performance Results for the Average Annual Daily Traffic Indicator,
                                 Compacts Ending in 2010 and 2011

                                     Country                   Original target    Revised target         Final result     Target met
                                                a
                                     Honduras                           3,772                                     N/A         N/A
                                     Cape Verde     b
                                                                          559                                     295          x
                                     Nicaragua                            932                                   1,216         ●
                                     Georgia                            6,700                1,183              1,092          x
                                     Vanuatu                              969                  196                334         ○
                                     Armenia                              460                  706                735         ●○
                                 Legend:
                                 ●      Met original target
                                 ○      Met revised target
                                 X      Did not meet original or revised target
                                 N/A Not available
                                 Source: GAO analysis of MCC data.

                                 Notes: A missing value for the revised target indicates that the original target was not revised. See
                                 appendix III for additional details on targets and results for the traffic indicator in each country.
                                 a
                                  The calculation for original and revised average annual daily traffic measures include the secondary
                                 roads and sections 3 and 4 of the highway in Honduras. Sections 1 and 2 of the highway were to be
                                 completed post-compact with additional funding provided by the Central American Bank for Economic
                                 Integration; as of March 9, 2012, construction of these segments was not complete. In addition, MCC
                                 officials stated that a final traffic measure in Honduras is not expected until late 2012.
                                 b
                                 The final result for Cape Verde is not annualized.




                                 20
                                   MCC revised the original traffic targets for the Georgia, Vanuatu, and Armenia
                                 compacts. In Georgia and Vanuatu, MCC reduced traffic targets when it corrected the
                                 original baseline estimates. In Armenia, MCC increased the traffic target once the road
                                 project was placed on indefinite hold; the single road that MCC completed prior to the
                                 project hold had a higher traffic target than the full package of roads MCC originally
                                 planned to complete.




                                 Page 16                                              GAO-12-631 MCC Transportation Infrastructure
                             MCC did not consistently update the baseline estimates for the annual
                             average daily traffic indicator either by projecting growth or by using new
                             information when it became available. Because MCC establishes the
                             targets for some indicators by applying estimated growth rates to the
                             baselines, these identified weaknesses may have affected MCC’s ability
                             to create accurate traffic targets, thus affecting its reported results.

                             •    For the secondary roads project in Nicaragua, MCC stated that it
                                  updated the baseline estimates established in 2007 by applying 1
                                  year’s growth rate to the traffic measures, because construction was
                                  expected to begin in fall 2008. However, MCC did not use this method
                                  in Cape Verde, where there was a similar time lag. 21

                             •    For the section of highway to be rehabilitated in Nicaragua, MCC
                                  retained a baseline calculated in a 2000 traffic study 22 even though it
                                  had 2007 data for the specific 18 kilometers of road it planned to
                                  rehabilitate.

                             •    In Honduras, MCC used a baseline established before 2005 despite
                                  the availability of data from a 2008 traffic survey. The 2008 traffic
                                  survey results showed that traffic volumes already met or exceeded
                                  the end-of-compact targets for most road segments. Because MCC
                                  used the 2005 data, the final results for the project indicate a larger
                                  change over the compact implementation timeframe than actually
                                  occurred.

Common Indicators for Road   The data quality reviews conducted for most compacts did not include the
Projects Not Consistently    kilometers-paved, roughness, or average annual daily traffic indicators.
Reviewed for Quality         MCC requires that MCAs contract an independent entity to review the
                             quality of compact performance data. According to MCC, this requirement
                             was fulfilled for every compact except Nicaragua. However, only the data
                             quality reviews for Georgia included all three indicators. (The reviews did
                             not identify any data problems with these indicators). 23 Although the data



                             21
                               MCC used a traffic count from a study conducted in 2005; however, construction of the
                             road project began in 2006.
                             22
                               In addition, the 2000 traffic study was conducted for a 72-kilometer section of highway,
                             rather than the 18 kilometers designated for the project.
                             23
                               MCC policy allows each country to determine the frequency and timing of each review,
                             along with which indicators will be evaluated.




                             Page 17                                      GAO-12-631 MCC Transportation Infrastructure
quality reviews for Armenia included two of the three indicators, the
compact was put on hold before MCA-Armenia could fully correct
identified problems. In Vanuatu, data quality reviews conducted in 2008
and 2009 identified problems with the measurement of annual average
daily traffic. 24 MCA-Vanuatu did not provide a response to either of the
recommendations in the review and made no changes to the definition of
the traffic indicator in the final monitoring and evaluation plan. The data
quality reviews for the Honduras and Cape Verde compacts did not
include any of the three road indicators. Table 5 summarizes the
treatment of the three common road indicators we examined in data
quality reviews for the six compacts.

Table 5: Data Quality Reviews Performed for Road Project Common Indicators

                      Kilometers of            International     Average
                          road                  roughness      annual daily     Recommendations
    Country            completed                   index          traffic       implemented
    Honduras                   ○                    ○                ○          N/A
    Cape Verde                 ○                    ○                ○          N/A
    Nicaraguaa                 X                    X                X          N/A
    Georgia                    ◓                    ◓                ◓          N/A
    Vanuatu                    ○                    ○                ●          No
    Armenia                    ○                    ●                ●          Nob

Legend:
○      Indicator not reviewed
◓      Indicator reviewed without recommendation
●      Indicator reviewed with recommendation
X      No data quality review performed
N/A Not applicable
Source: GAO analysis of MCA data quality reviews.
a
MCC officials stated that no data quality reviews were performed in Nicaragua.
b
 Recommendations were not implemented in Armenia, because the road project was put on hold in
June 2009.




24
  The 2008 review provided several recommendations on how to structure the traffic count
survey. In addition, a 2009 review found that the Vanuatu compact was at risk of double
counting vehicles and recommended that the monitoring and evaluation plan indicate that
traffic refers to the sum of the average number of vehicles passing over checkpoints.




Page 18                                                   GAO-12-631 MCC Transportation Infrastructure
MCC Did Not Consistently   The governments of three countries—Honduras, Cape Verde, and
Account for Third-Party    Vanuatu—secured funding from a third party to rehabilitate some of the
Funding                    road lengths that MCC removed from the scope of the compact. 25
                           However, we found that MCC did not consistently account for this third-
                           party funding when reporting the final results of kilometers completed. As
                           a result, MCC understated its results in Honduras and overstated them in
                           Vanuatu.

                           •     The government of Honduras secured money from the Central
                                 American Bank for Economic Integration to complete 57.6 kilometers
                                 of the highway project, which MCC removed from the compact’s
                                 original scope. 26 However, MCA-Honduras did not revise its
                                 kilometers-completed target to reflect this reduction in scope.
                                 Therefore, MCC reports that it did not meet the compact’s final
                                 kilometers-completed target, even though data suggest that the
                                 kilometers MCC funded were complete at the end of the compact

                           •     The government of Cape Verde secured funding from the government
                                 of Portugal to rehabilitate 22.9 kilometers of secondary roads that
                                 MCC removed from the scope of its compact. MCA-Cape Verde
                                 revised its kilometers-completed target to reflect this reduction in
                                 scope and did not report these 22.9 kilometers as a part of its final
                                 results. Therefore, MCC reports that it met the compact’s revised
                                 kilometers-completed target.

                           •     The government of Vanuatu secured money from New Zealand to
                                 assist in rehabilitating a 57.2 kilometer road, because the MCC
                                 compact funds were not sufficient to complete all 57.2 kilometers.
                                 However, MCC reported all 57.2 kilometers as a part of its final
                                 results, even though it did not fund all of the work.




                           25
                               Third-party funding is funding from a source other than MCC or the partner government.
                           26
                             MCC originally planned to fund all reconstruction included in the four sections of the
                           Honduras highway project. However, because estimated costs had increased, MCC
                           determined that it would be unable to complete all four sections within the 5-year compact
                           timeframe and within the funding allocation. As a result, MCC funded about 94 percent of
                           construction on sections 3 and 4, funded 31 percent of section 2, and funded none of
                           section 1. The third-party funding from the Central American Bank of Economic Integration
                           funded the difference. For more information on this project, see GAO-11-728.




                           Page 19                                       GAO-12-631 MCC Transportation Infrastructure
MCC Reduced the Cape           For the Cape Verde compact, MCC reduced the scope of the port project,
Verde Port Project Scope       deferred the measurement of key indicators, and therefore did not
and Did Not Meet Many          conduct any data quality reviews of those indicators. In Benin, MCC
                               completed most of the originally envisioned port project scope, and its
Targets for the Benin Port     data show that the compact met the original target for one of the three
Project                        key indicators. However, we identified issues with the indicators’ baseline
                               estimates. In addition, data quality reviews identified problems with the
                               data behind the indicators, which MCA-Benin did not formally address.

In Cape Verde, MCC Reduced     MCC split the Cape Verde project into two phases in May 2008 because,
the Port Project’s Scope and   owing to increased construction costs, the $53.7 million allocated to the
Deferred the Measurement of    port projects was insufficient to fund all of the planned improvements.
Key Indicators                 MCC funded phase 1 of the improvements, including construction of a
                               cargo storage area and an access road, and rehabilitation of Wharf 2.
                               The government of Cape Verde secured a loan from the government of
                               Portugal for $87 million to complete phase 2, that is, those elements
                               removed from the scope of the MCC project. 27 The MCC-funded portion
                               of the construction (phase 1) represents about one-third of the total
                               expected cost of both phases. Nearly 100 percent of the works for phase
                               1 were completed by compact end. Phase 2 was awarded late in the
                               compact, and work was under way as of July 2012.

                               MCC deferred the measurement of key performance indicators when it
                               revised the project scope, because results for key indicators could be
                               measured only after full completion of the port (see table 6). MCC intends
                               to measure results for the key indicators when phase 2 is complete.

                               Table 6: Key Performance Results for MCC Port Project in Cape Verde

                                                                                    Original      Revised        Final      Target
                                Indicator                                             target       target        result      met
                                Volume of merchandise (thousand                        710.5
                                tons/year)                                                        Performance indicators were
                                                                                                   eliminated once the Cape
                                Container ship time at berth (days)                       1.0
                                                                                                  Verde project was descoped.
                                Container ship time at anchor (hours)                     4.0
                               Source: GAO analysis of MCC data.

                               Note: See appendix IV for additional details on targets and results for key indicators in Cape Verde.




                               27
                                 Phase 2 works included the expansion of Wharf 1 and construction of a new breakwater.




                               Page 20                                            GAO-12-631 MCC Transportation Infrastructure
                                 MCA-Cape Verde contracted for data quality reviews in 2007 and 2010.
                                 The port project was reviewed in 2010; however, key indicators were not
                                 evaluated, because MCA-Cape Verde deferred measurement of those
                                 indicators until phase 2 was complete.

MCC’s Reported Results for the   MCC data show that Benin maintained most of the original project scope
Benin Port Project Have Data     and met the original target for one of three key indicators. 28 The project
Quality Issues                   scope included constructing or rehabilitating a jetty, a new wharf, and port
                                 roads and installing lighting, security, and an electricity distribution
                                 system. MCC reports that the Benin compact met the original target for
                                 volume of merchandise through the port, but it did not meet targets for the
                                 other two key indicators, container ship waiting time at berth and
                                 container ship waiting time at anchor. See table 7 for a summary of key
                                 performance results for the Benin port project.

                                 Table 7: Key Performance Results for MCC Port Project in Benin

                                                                                          Original      Revised         Final Target
                                  Indicator                                                 target        target       result  met
                                  Volume of merchandise (thousand tons/year)                6,944.6                  7,605.9        ●
                                  Container ship time at berth (days)                            1.0                       1.3      x
                                  Container ship time at anchor (hours)                          4.0                     34.6       x
                                 Legend:
                                 ●   Met original target
                                 ○   Met revised target
                                 x   Did not meet original or revised target
                                 Source: GAO analysis of MCC data.

                                 Note: A missing value for the revised target indicates that the original target was not revised. See
                                 appendix IV for additional details on targets and results for key indicators in Benin.




                                 28
                                    While the project was largely completed as planned, components for a proposed fourth
                                 lot—which included a storage facility for dry bulk goods such as grains and sand and a
                                 fish quality inspection station—were deemed not viable once MCA-Benin conducted its
                                 feasibility studies. As a result, MCA-Benin did not tender a bid for that lot. According to
                                 MCC officials, the funds originally planned for those items were shifted to the other
                                 infrastructure components. The funds also helped cover cost increases and additional
                                 work on the wharf such as increasing wall length and dredging the berth.




                                 Page 21                                             GAO-12-631 MCC Transportation Infrastructure
We identified problems with MCC’s measurement of the volume-of-
merchandise indicators, and MCA-Benin’s data quality reviews identified
additional problems with all three key indicators. For the volume-of-
merchandise indicator in Benin, MCC used 2004 data from the Port
Authority to establish a baseline of 4.1 million metric tons in the compact’s
monitoring and evaluation plans and indicator tracking tables. MCA-Benin
did not subsequently update the baseline. However, the compact did not
enter into force until October 2006 and port improvement construction did
not begin until August 2009. Available annual data for Benin showed that
in 2005 the volume of merchandise had already increased to 5.2 million
metric tons. Without an updated baseline, the estimate may not
accurately reflect preconstruction conditions and may lead MCC to
overstate the degree to which port traffic increased because of the
compact.

MCA-Benin contracted data quality reviews in 2008, 2009, and 2011. The
2008 and 2009 data quality reviews identified issues with the three key
port indicators that we reviewed, but MCC did not address all of the
issues. 29 The 2011 review did not examine these specific port indicators.

•   For the volume-of-merchandise indicator, MCA-contractors noted that
    although data collection methods were consistent and the data were
    valid, the Port Authority of Cotonou needed to improve the timeliness
    of data related to this indicator because the data entry and control
    procedures created delays in the availability of information. MCC
    policy indicates that data should be sufficiently current to inform
    management decisions because decisions depend on regular
    collection of performance information. In 2008 and 2009, the
    contractors recommended that MCA-Benin set up an automatic data
    transfer system for information collected on ship stopover, to avoid
    redundant data entry efforts by the port’s statistical services. Although
    the 2009 review indicated that the port was resolving problems



29
  Reviews also identified other port indicators for improvement in Benin. For example, the
2008 and 2009 reviews indicated that the average duration of stay of trucks at port, which
measures the reduction in time necessary for vehicles to cross the port due to reduced
congestion, had a baseline value that was set with no justification of its actual level. The
review recommended that this information be collected in the annual survey of port user
satisfaction. MCA-Benin did so and determined that the baseline value was 7 hours.
However, no changes were made to the final monitoring and evaluation plan to reflect this
and MCC and MCA officials indicated that they agreed not to revise indicator baselines at
the time.




Page 22                                       GAO-12-631 MCC Transportation Infrastructure
                           through the creation of a port information system when the evaluation
                           was conducted, we found that MCA-Benin did not resolve the issues
                           identified for this indicator. No changes were made in the final
                           monitoring and evaluation plan to indicate that adjustments were
                           made in the data collection methods for this indicator. In addition,
                           MCA-Benin responses to the review did not address this issue.

                       •   For the container ship time at berth and container ship time at anchor
                           indicators, the 2008 and 2009 reviews noted that, while the data
                           collected were a direct measure of what the indicators were intended
                           to capture and the data collection procedures were consistent, there
                           were no historical data to support the baseline values for these
                           indicators. The reviews recommended using baseline values
                           established in a 2005 report by an international consulting firm. We
                           found that MCA-Benin did not resolve issues identified for these two
                           indicators. MCC and MCA officials stated that they decided not to
                           revise indicator baselines and targets as recommended, and no
                           revisions were made to baseline figures in subsequent monitoring and
                           evaluation plans. MCC and MCA officials provided no additional
                           explanation.


                       In 2009, MCC improved its beneficiary estimates by adopting a more
Data Resulting from    standardized approach, which it documented in its Guidelines for
Beneficiary            Economic and Beneficiary Analysis; however, the updated guidelines do
                       not include a formal quality review process. MCC subsequently revised its
Calculations May Not   beneficiary estimates for ongoing compacts, but the implementation of the
Be Valid               new approach was not always consistent and suffered from varying
                       degrees of quality control problems. Although some of the individual
                       problems we identified were small in nature, taken as a whole they
                       reduce confidence in MCC’s estimates.




                       Page 23                              GAO-12-631 MCC Transportation Infrastructure
MCC’s Updated                    In our previous work, we found that MCC did not have a consistent
Beneficiary Analysis             methodology for estimating the number of beneficiaries across
Guidelines Do Not Include        compacts. 30 As a result, MCC adopted a new, more standardized
                                 approach, which it published in its 2009 Guidelines for Economic and
a Formal Quality Review
                                 Beneficiary Analysis. MCC subsequently conducted a “beneficiary scrub,”
Process                          revising beneficiary numbers for ongoing compacts based on the 2009
                                 guidelines. However, we found errors and inconsistencies in the
                                 calculations. MCC stated that these scrubs are informally reviewed but
                                 that they do not undergo a formal quality review process. 31 Such a
                                 process would ensure that the information and data provided have been
                                 verified for consistency and accuracy.


Weaknesses Call into             We identified three general weaknesses with MCC’s beneficiary
Question the Validity and        calculations. First, the beneficiary calculations in MCC’s internal
Reliability of MCC’s             documents contain weaknesses such as incorrect formulas and numbers
                                 as well as discrepancies with supporting documents. Second, MCC did
Beneficiary Data                 not apply a consistent methodology in estimating beneficiaries for its early
                                 transportation infrastructure projects. Third, the beneficiary figures in
                                 MCC’s public documents are sometimes inaccurate.

MCC’s Beneficiary Calculations   Some beneficiary calculations contained mistakes in the formulas used.
Contained Mistakes               For example, the computations of the population growth rate for Georgia
                                 and Cape Verde contained an error in the mathematical formula, causing
                                 MCC to slightly overstate the growth rate in the first case and understate
                                 it in the second case. In addition, erroneous numbers were used in four of
                                 the seven compacts we reviewed.

                                 •    For the Georgia road project, MCC stated in the beneficiary
                                      calculation documents that “the original beneficiary estimate of 53,988
                                      people counts all households living in the four rayons, or districts,



                                 30
                                   See GAO, Millennium Challenge Corporation: Independent Review and Consistent
                                 Approaches Will Strengthen Projections of Program Impact, GAO-08-730 (Washington,
                                 D.C.: Jun. 17, 2008).
                                 31
                                   According to Standards for Internal Control in the Federal Government, a variety of
                                 control activities should be used in information processing, including checking the data
                                 entered. In addition, GAO’s Internal Control Management and Evaluation Tool states that
                                 data validation and editing should be performed to identify erroneous data, which need to
                                 be reported and promptly corrected (GAO, Internal Control Management and Evaluation
                                 Tool, GAO-01-1008G (Washington, D.C.: August 2001).




                                 Page 24                                     GAO-12-631 MCC Transportation Infrastructure
                                     through which the road passes…The revised beneficiary estimate [of
                                     58,079] is based on a 5 km catchment area of the Road.” MCC thus
                                     originally claimed a lower number of beneficiaries for a larger
                                     geographic area (the districts) than the more limited catchment area—
                                     the geographic area in which benefits may be expected to accrue—of
                                     5 kilometers on either side of the road. 32

                                •    For the Benin port project, the population projection in the beneficiary
                                     scrub began with a 2005 original baseline figure of 8,490,000, since
                                     the port project is expected to raise the incomes of the entire
                                     population of Benin. However, this number could not be derived from
                                     its purported source, the 2002 population census. 33

                                •    For the Armenia road project, the document supporting the beneficiary
                                     scrub calculation considered a catchment area for an erroneous
                                     revised road length of 68 kilometers rather than the correct figure of
                                     24.4 kilometers.

                                •    In the Cape Verde roads and bridges activity, the original beneficiary
                                     numbers provided in the document supporting the beneficiary scrub
                                     spreadsheet did not match the figures in the spreadsheet.

MCC’s Methodology to            In the 2009 revision of its Guidelines for Economic and Beneficiary
Estimate Beneficiaries Is Not   Analysis, MCC standardized its methodology to estimate beneficiaries
Consistently Applied across     and accordingly modified its beneficiary numbers in the spreadsheets for
Compacts                        its beneficiary scrubs. However, the amount of detail in the analysis
                                varied across compacts, and population growth rates were not always
                                calculated in a consistent manner.

                                Of the beneficiary scrubs we reviewed, four of seven contained
                                disaggregated data at the project level. Specifically, the Armenia,
                                Georgia, Nicaragua, and Vanuatu scrubs are made up of multiple,
                                relatively detailed spreadsheets. By contrast, one beneficiary scrub, for
                                Honduras, contained one page of macro-level summary data with no



                                32
                                  MCC officials told us that the original number of 53,988 beneficiaries was erroneous and
                                should have been 111,442, in accordance with the 2004 Georgian Agricultural Census.
                                33
                                  The 2002 census indicates 6,769,914 for the population of Benin; adjusting population
                                growth for 3 years leads to 7,419,235, not 8,490,000, in 2005. In addition, the supporting
                                document that accompanied the beneficiary scrub spreadsheet and explained the
                                computations had a different “original” baseline number—8,791,832—for 2005.




                                Page 25                                      GAO-12-631 MCC Transportation Infrastructure
                              explanation or sources for the figures. The remaining two of the seven
                              scrubs contained additional data but were not as detailed as the first four.
                              For example, the Cape Verde scrub included a second sheet of World
                              Bank development indicators on the country, and the Benin scrub had
                              some details on one of the compact projects. These varying levels of
                              detail suggest that the scrubs were executed in an ad hoc, rather than
                              consistent, manner.

                              MCC uses population growth rates to estimate the number of
                              beneficiaries in a 20-year projection period; for each compact, MCC
                              publicly presents the number of beneficiaries projected for the end of the
                              20-year period. 34 However, the scrub sheets we reviewed showed that
                              MCC sometimes computed population growth rates differently across
                              compacts, even when the scrubs provided the same underlying
                              information.

                              •    For Honduras, MCC indicated that it took the population growth rates
                                   for 2005, 2006, and 2007 from World Bank data and averaged the
                                   three rates to get an average annual growth rate for the projection. 35

                              •    For Georgia and Cape Verde, MCC did not use the historical growth
                                   rates provided but instead took total population figures in 2005 and
                                   2007 and used them to compute an average annual growth rate for
                                   the beneficiary projection.

                              •    For Vanuatu, MCC computed an average annual growth rate based
                                   on four historical growth rates without specifying a source or year.

For Some Countries, MCC Did   In three of seven compacts, we found that some numbers in MCC’s public
Not Update Beneficiary        documents had not been updated or were inaccurate.
Numbers or Reported Them
Incorrectly                   •    In the case of the Georgia road, a December 2009 report and a July
                                   2011 close-out document used original beneficiary estimates
                                   (53,988), instead of using the revised estimate (58,079) from July
                                   2009, when MCC reported having recalculated the beneficiary
                                   numbers. In addition, in the 2011 status report, the total number of


                              34
                                MCC officials indicated that the standard time horizon for the beneficiary analysis is
                              generally 20 years but that exceptions exist.
                              35
                                In the case of Benin and Nicaragua, the growth rate for the year 2000 was added to the
                              formula.




                              Page 26                                       GAO-12-631 MCC Transportation Infrastructure
                  beneficiaries for the compact as a whole (143,000) was smaller than
                  the beneficiaries of the infrastructure project alone (300,000).

              •   For the Benin port project, the beneficiary estimate that appeared in a
                  2011 status report (8,791,832) was an original estimate from 2009
                  instead of the revised estimate of 16,119,058. In addition, the
                  supporting document accompanying the 2009 scrub spreadsheet
                  mistakenly reported 13,421,086 beneficiaries for the port project by
                  2026. However, the data in the scrub indicated that this figure referred
                  to the number of beneficiaries in 2020; the correct number of
                  projected beneficiaries for 2026 is 16,119,058.

              •   For Armenia, the beneficiary estimate for the road project in a 2011
                  publication (6,216) did not reflect a revision of the estimate done in
                  2009 (6,356).


              MCC recognizes the importance of a disciplined, transparent, and
Conclusions   accountable approach to tracking compact results to make well-informed
              decisions about U.S. investments. It has established an extensive
              monitoring and evaluation program that includes guidance and tools for
              the partner countries and controls on data quality. In addition, MCC has
              created and implemented a revised beneficiary calculation methodology
              to ensure standardization across projects and compacts.

              Even with these steps, problems exist with the data behind several of
              MCC’s key performance indicators, limiting the reliability of MCC’s
              reported results.

              •   MCAs have not measured the results of common indicators in a
                  uniform manner—including how they account for additional third-party
                  funding—nor have they consistently reviewed the quality of data used
                  to report the indicators’ results. As such, MCC cannot confidently
                  compare or aggregate the results of these common indicators as
                  intended.

              •   MCC requires the establishment of baselines to measure the impact
                  its assistance has on partner countries. However, because some
                  incorrect baselines were used, results may be over- or understated.

              •   MCAs have not consistently fulfilled the MCC requirements that they
                  conduct independent data quality reviews and respond to issues the
                  reviews raised. As a result, MCC and its partners have missed



              Page 27                               GAO-12-631 MCC Transportation Infrastructure
                          opportunities to improve data quality and facilitate data reporting and
                          aggregation.

                      In addition, MCC’s beneficiary calculations have problems that limit their
                      usefulness. Because MCC’s new approach to estimating beneficiary
                      numbers does not include a formal quality review process, the beneficiary
                      estimates contain incorrect numbers and formulas and inconsistent
                      calculations. While the impact of these problems is generally small, the
                      lack of a robust, formal quality review process nonetheless calls into
                      question the reliability of MCC’s estimates. Without a correct
                      representation of the compacts’ results and beneficiaries, MCC,
                      Congress, and other key stakeholders cannot accurately evaluate the
                      extent to which MCC is achieving its goals of poverty reduction and
                      economic growth.


                      We recommend that MCC’s Chief Executive Officer take the following
Recommendations for   four actions.
Executive Action
                      To improve the reliability of results measurement, MCC should

                      •   improve guidance for common indicators by requiring that they are
                          included in data quality reviews, measured uniformly across
                          compacts, and reported in a standardized manner—including when
                          third-party funding is used to complete the original scope of a project;

                      •   improve guidance for baseline measurements by requiring MCAs to
                          document the date, source, and methodology for establishing the
                          baseline; and

                      •   enforce current monitoring and evaluation policy requiring MCAs to
                          conduct data quality reviews and to provide written comments in
                          response to identified issues, discussing how recommendations will
                          be implemented or explaining why changes may not be made.

                      To ensure more accurate beneficiary numbers, MCC should

                      •   incorporate into the Guidelines for Economic and Beneficiary Analysis
                          a formal process for reviewing beneficiary calculations and analysis.




                      Page 28                               GAO-12-631 MCC Transportation Infrastructure
                     In written comments on a draft of this report, MCC stated that it agrees
Agency Comments      with our four recommendations and outlined steps it will take or has taken
and Our Evaluation   to address them.

                     •   With respect to our first recommendation—to improve guidance for
                         common indicators—MCC stated that it has taken preliminary actions
                         to address it. We reviewed MCC’s May 2012 Guidance on Common
                         Indicators and found that it defines the indicators and specifies each
                         indicator’s unit of measurement and how each should be
                         disaggregated. In addition, when MCC activities are conducted jointly
                         with other organizations (e.g. when the activities include third-party
                         funding), MCC’s guidance instructs MCAs to report only MCC’s
                         contribution to a particular common indicator. In addition, the
                         guidance states that MCC will be explicit in reporting which changes
                         in outcome indicators are more likely the result of MCC investments
                         and which changes might also be influenced by other interventions.
                         However, the guidance lacks enough specificity to address other
                         weaknesses we identified. For example, when discussing road
                         roughness, the guidance does not instruct MCAs to measure both
                         sides of the road to account for variation in roughness from one side
                         of the road to the other. In addition, this guidance does not specify
                         that MCAs should include common indicators in their data quality
                         reviews.

                     •   With respect to our second recommendation—to improve guidance for
                         baseline measurements—MCC stated that its current monitoring and
                         evaluation policy requires that compacts’ monitoring and evaluation
                         plans document the date, source, and methodology for establishing
                         baseline measurements. MCC further stated that current MCA
                         monitoring and evaluation plans are compliant with this requirement.

                     •   With respect to our third recommendation—to enforce current
                         monitoring and evaluation policy requiring MCAs to conduct data
                         quality reviews and to provide written comments in response to
                         identified issues—MCC stated that it works with MCAs to ensure that
                         recommendations of data quality reviews are incorporated into
                         monitoring and evaluation plans and then implemented.

                     •   With respect to our fourth recommendation—to incorporate a formal
                         process for reviewing beneficiary calculations and analysis—MCC
                         noted that it will institute a formal process for reviewing beneficiary
                         calculations and address the weaknesses we identified in this report.



                     Page 29                               GAO-12-631 MCC Transportation Infrastructure
MCC asserted that the results of its compact investments are not
diminished by the data quality challenges highlighted in this report. MCC
noted, for example, that more than 623 kilometers of rehabilitated roads
now serve poor and rural households in the countries we reviewed,
linking farmers to markets and bolstering important regional trade routes.
Regarding our finding that the total target for kilometers of road
completed for the six compacts was reduced by 63 percent, MCC also
noted that the final target for Honduras, Georgia, Cape Verde, and
Vanuatu would be only 19 percent below the original target if we were to
exclude the road projects that MCC terminated or discontinued as a result
of the Armenian and Nicaraguan governments’ undemocratic behavior.

We have reprinted MCC’s comments in appendix V. We have also
incorporated technical comments from MCC in our report where
appropriate.


We are sending copies of this report to interested congressional
committees and the Millennium Challenge Corporation. In addition, this
report will be available at no charge on the GAO Web site at
http://www.gao.gov.

If you or your staff has any questions about this report, please contact
David Gootnick at (202) 512-3149 or gootnickd@gao.gov. Contact points
for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. GAO staff who made major
contributions to this report are listed in appendix VI.




David Gootnick
Director
International Affairs and Trade




Page 30                             GAO-12-631 MCC Transportation Infrastructure
Appendix I: Objectives, Scope, and
              Appendix I: Objectives, Scope, and
              Methodology



Methodology

              The fiscal year 2008 Consolidated Appropriations Act, Public Law 110-
              161, mandated that GAO review the results of the Millennium Challenge
              Corporation’s (MCC) compacts. This report examines the extent to which
              MCC has, for transportation infrastructure projects in compacts ending in
              2010 and 2011, (1) achieved expected performance targets and (2) used
              a consistent methodology in estimating numbers of beneficiaries. The
              seven countries with compacts relevant to our scope are Honduras, Cape
              Verde, Nicaragua, Georgia, Vanuatu, Armenia, and Benin.

              MCC enters into a legal relationship with partner country governments,
              which vests responsibility for day-to-day management of compact project
              implementation with an accountable entity—a Millennium Challenge
              Account (MCA)—including monitoring and evaluation activities such as
              setting and revising targets. Because such MCA actions require MCC’s
              direct oversight and approval, throughout this report we attribute all
              decisions related to project rescoping and compact targets to MCC.

              To assess the extent to which MCC has achieved its performance targets
              for transportation infrastructure projects in compacts ending in 2010 and
              2011, we reviewed MCC guidance and policy documents and analyzed
              compacts, monitoring and evaluation plans, and indicator tracking tables
              for the compacts with Honduras, Cape Verde, Nicaragua, Georgia,
              Vanuatu, Armenia, and Benin. In assessing and reporting MCC’s results,
              we compared actual results achieved at the end of the compact for select
              performance indicators with the original and, in some cases, revised
              targets associated with each indicator. We considered the original target
              to be the one first documented for each performance indicator and the
              final target to be the one last documented in MCC monitoring documents.

              Given that MCC tracks several performance indicators for each compact
              project, we selected a subset of indicators to examine for this report. For
              the roads projects, we examined kilometers of road completed, road
              roughness, and average annual daily traffic. We selected these road
              project indicators because (1) they address the projects’ key objectives
              and (2) they are among the “common indicators” that MCC requires all
              road projects to measure so that it can aggregate results across
              countries. For the port projects, we examined volume of merchandise
              through the port, container ship time at berth, and container ship time at
              anchor. Although MCC has not established common indicators for the
              port projects, we selected these port project indicators because they (1)
              addressed the projects’ key objectives, (2) corresponded to the common
              indicators established for the roads projects, and (3) were originally
              included in both port projects.


              Page 31                               GAO-12-631 MCC Transportation Infrastructure
Appendix I: Objectives, Scope, and
Methodology




To report the average value of targets and results for compact road
projects, we calculated a weighted average using the length of each road
segment. We performed this calculation for the average annual daily
traffic and roughness indicators for Cape Verde, Honduras, Vanuatu, and
Nicaragua because these MCAs report targets and results by road
segment. For Honduras, we did not weight the indicators using values
from sections 1 and 2 because the government of Honduras completed
these sections with funding from the Central American Bank for Economic
Integration. For Cape Verde, we did not include the average annual daily
traffic and roughness for bridge projects, because the kilometer lengths of
these sections were not available.

To determine the amount of funding used for transportation infrastructure
projects, we reviewed MCC financial data. We included compact
implementation funding—funds disbursed before entry into force to
facilitate the implementation of the compact—with other projects not
related to transportation infrastructure.

To determine which MCAs contracted for independent data quality
reviews, which transportation infrastructure indicators were evaluated,
and whether recommendations were made for common and key
indicators, we reviewed third-party data quality review reports. To
determine whether recommendations were implemented, we reviewed
MCA comments on the data quality reviews that MCC provided to us and
we reviewed final monitoring and evaluation plans.

To assess the extent to which MCC has consistently applied its
beneficiary estimation methodology, we reviewed MCC’s beneficiary
calculations and accompanying supporting documents for the seven
compacts that ended in 2010 and 2011. In particular, we examined for
consistency and accuracy the data and formulas on projected
beneficiaries for road and port projects in the Excel spreadsheets that
MCC used to update—or “scrub”—the beneficiary estimates. We
reviewed the documents explaining the calculations. We compared
original beneficiary figures for these transportation projects with the
revised ones and assessed the justification behind the changes. We also
examined quarterly status reports and the monitoring and evaluation
plans for each compact to check the accuracy of the publicly reported
beneficiary counts. We consulted with MCC officials when we found
discrepancies and errors in the data.




Page 32                              GAO-12-631 MCC Transportation Infrastructure
Appendix I: Objectives, Scope, and
Methodology




As part of related GAO engagements focusing on Cape Verde and
Honduras 1 and on Georgia and Benin, 2 we visited compact projects in-
country and met with MCA officials, partner country government officials,
contractors, project managers, construction supervisors, and relevant
private businesses.

Data are considered reliable when they are accurate, complete,
consistent and valid, given the uses for which they are intended. To
assess the reliability of MCC indicator and beneficiary data, we (1)
reviewed MCC policy for the monitoring and evaluation of compacts and
for calculating beneficiary numbers, (2) interviewed MCC and MCA
officials regarding the reliability and validity of the data, (3) reviewed
independent data quality reviews required by MCC, (4) compared
performance indicators across original and final monitoring and evaluation
plans and indicator tracking tables to identify inconsistencies, and (5)
reviewed beneficiary calculations for accuracy. We identified some
weaknesses in the data that may affect the data’s reliability, as discussed
in this report.

Finally, some of the reports and documents referenced above were
written in French and Spanish. We translated these documents as
needed, creating English summaries to enable our analysis.

We conducted this performance audit from August 2011 to September
2012 in accordance with generally accepted government auditing
standards. 3 Those standards require that we plan and perform the audit
to obtain sufficient, appropriate evidence to provide a reasonable basis
for our findings and conclusions based on our audit objectives. We
believe the evidence obtained provides a reasonable basis for our
findings and conclusions based on our audit objectives.




1
GAO-11-728.
2
GAO-12-630.
3
 We began work for this report in 2010; however, the job was suspended while we
conducted work for a related engagement.




Page 33                                    GAO-12-631 MCC Transportation Infrastructure
Appendix II: Compact Timeframes
             Appendix II: Compact Timeframes




             The Millennium Challenge Corporation (MCC) completed seven compacts
             in 2010 and 2011—Honduras, Cape Verde, Nicaragua, Georgia, Benin,
             Vanuatu, and Armenia (see fig. 3). MCC commits funding when it signs a
             compact, but MCC does not obligate funding until the compact enters into
             force. MCC funding that the partner country does not spend for compact
             activities by the end of the 5-year implementation timeframe must be
             deobligated. 1




             1
              The statutory 5-year maximum timeframe begins once a compact enters into force.




             Page 34                                   GAO-12-631 MCC Transportation Infrastructure
Appendix II: Compact Timeframes




Figure 3: Timeframes for MCC Compacts Ending in 2010 and 2011, by Date of
Signature




Page 35                                GAO-12-631 MCC Transportation Infrastructure
Appendix III: Roads Data
                                          Appendix III: Roads Data




Table 8: Kilometers of Roads Completed for MCC Road Projects in Compacts Ending in 2010 and 2011

                                                                                                     Percentage of       Percentage of final
Country             Original target             Final target                  Final result      original target met              target met
Honduras                       200.0                    174.5a                      115.0                      57.5                        65.9
Cape Verde                      63.0                       39.3                          40.6                  64.4                       103.3
Nicaragua                      158.0                       67.0                          74.0                  46.8                       110.4
Georgia                        245.0                     220.2                      220.0                      89.8                       100.0
Vanuatu                        213.8                     149.7                      149.7                      70.0                       100.0
Armeniab                       943.0                       24.4                          24.4                   2.6                       100.0
                                          Source: GAO analysis of MCC data.
                                          a
                                           After reducing the scope of the Honduras compact, MCC formally reduced the final target for the
                                          secondary road activities but did not reduce the final target for the highway activity.
                                          b
                                           MCC provided documentation of kilometers completed; however, the documentation did not include
                                          takeover certificates from contractors, the data source for final kilometers in other countries.



Table 9: Road Roughness of MCC Road Projects in Compacts ending in 2010 and 2011, as Measured by the International
Roughness Index (IRI)

                                                     Roughness                Original                                       IRI
                                                                                                                                        a
Country      Road section                              baseline                 target      Final target    Final result     methodology
Hondurasb
             CA-5 Highway: Section 1                              4.7              1.9               1.9              N/C
             CA-5 Highway: Section 2                              4.4              1.9               1.9              N/C
             CA-5 Highway: Section 3                              4.0              1.9               1.9               2.2   Mechanical
             CA-5 Highway: Section 4                              4.0              1.9               1.9               3.2   Mechanical
             Secondary roads                                   13.6                2.5               2.5               3.2   Mechanical
Cape Verde
             Road 1: Orgãos-Pedra Badejo                       19.0                2.0               2.0               2.0   Visual
             Road 2: Cruz Grande-Calhetona                     18.0                2.0               2.0               2.0   Visual
             Road 4: Assomada-Rincão                           20.7                2.0               2.0               2.0   Visual
             Road: Vila das Pombas-Eito                        18.0              10.0               10.0              10.0   Visual
Nicaragua
             Highway N-1: Villanueva-                          12.0                3.4               3.4               1.8   Mechanical
             Guasaule
             S1: Somotillo-Cinco Pinos                         13.2                3.0               3.0               3.4   Mechanical
             S9: Leon-Poneloya                                 12.0                3.0               3.0               1.8   Mechanical
Georgia                                                        16.6                3.2               2.5               1.5   Mechanical




                                          Page 36                                               GAO-12-631 MCC Transportation Infrastructure
                                                 Appendix III: Roads Data




                                                            Roughness                 Original                                      IRI
                                                                                                                                               a
Country            Road section                               baseline                  target        Final target   Final result   methodology
Vanuatu
                   Efate Ring Road                                    17.5                   3.5              3.5             3.0   Visual
                   Santo East Coast Road                              22.0                   3.5              3.5             3.0   Visual
Armenia                                                               14.2                   5.0              4.0             3.5   Mechanical
                                                 Source: GAO analysis of MCC data.

                                                 N/C = not complete.
                                                 Note: We found weaknesses in some of the reported data on road roughness, including the use of
                                                 inconsistent application of methodologies and calculation errors.
                                                 a
                                                  There are two standard methods for measuring IRI: mechanical reading and visual observation.
                                                 Mechanical reading involves using a device such as a bump integrator, walking profiler, or car-
                                                 mounted software equipment to measure the IRI for each kilometer of road, which is then averaged
                                                 across the length of the road to provide a single IRI estimate. The margin of error for mechanical
                                                 readings differs depending on the type of device used. Visual observation is based on interpreting the
                                                 smoothness of an observers’ drive of the road at different speeds in a standard passenger vehicle, as
                                                 corresponding to an IRI visual assessment scale. The accuracy of the visual inspection varies with
                                                 the experience of the observer.
                                                 b
                                                  Because of the rescoping of the Honduras compact, sections 1 and 2 of the CA-5 Highway were
                                                 completed post-compact with additional funding provided by the Central American Bank for Economic
                                                 Integration; as of March 9, 2012, construction of these segments was not complete.



Table 10: Average Annual Daily Traffic on MCC-Funded Roads in Compacts Ending in 2010 and 2011

                                                                      Average annual
                                                                          daily traffic
Country                                                                      baseline              Original target      Final target     Final result
           a
Honduras
                    CA-5 Highway: Section 1                                          8,374                  9,447             9,447               N/C
                    CA-5 Highway: Section 2                                          5,411                  6,104             6,104               N/C
                    CA-5 Highway: Sections 3 & 4                                     6,732                  7,594             7,594               N/C
                    Secondary roads                                                   676                     887               887               N/C
               b
Cape Verde
                    Road 1: Orgãos-Pedra Badejo                                       298                     375               375               432
                    Road 2: Cruz Grande-Calhetona                                     440                     554               554               235
                    Road 4: Assomada-Rincão                                           543                     684               684               268
                    Road: Vila das Pombas-Eito                                        410                     517               517               184
                    Bridge Ribeira Grande-Paul                                        552                     695               695                76
                    Bridge Ribeira da Torre-Ponta de Sol                              765                     963               963               644
                    Bridge A Paul (Vila das Pombas)                                   671                     845               845                50




                                                 Page 37                                                 GAO-12-631 MCC Transportation Infrastructure
                                        Appendix III: Roads Data




                                                             Average annual
                                                                 daily traffic
Country                                                             baseline        Original target          Final target      Final result
Nicaragua
            Highway N-1: Villanueva-Guasaule                                1,413             1,580                 1,580              1,962
            S1: Somotillo-Cinco Pinos                                        234                278                   278                561
            S9: Leon-Poneloya                                               1,103             1,276                 1,276              1,462
Georgia                                                                      612              6,700                 1,183              1,092
Vanuatu
            Efate Ring Road                                                   85                950                     98               305
            Santo East Coast Road                                            307              1,000                   355                381
Armenia                                                                      637                460                   706                735
                                        Source: GAO analysis of MCC data.

                                        N/C = not complete.
                                        Note: We found weaknesses in some of the reported data on average annual daily traffic, including
                                        inconsistent updating of baseline measurements.
                                        a
                                         A traffic count for completed roads in Honduras was conducted from December 19, 2010, through
                                        January 25, 2011. MCC stated that it does not expect a final traffic measure in Honduras until late
                                        2012.
                                        b
                                         As of January 2012, a final traffic study for completed roads in Cape Verde had not been conducted;
                                        results shown reflect traffic counts conducted at the compact end date and do not reflect annualized
                                        average daily traffic volumes.




                                        Page 38                                           GAO-12-631 MCC Transportation Infrastructure
Appendix IV: Port Data
                                        Appendix IV: Port Data




Table 11: Volume of Merchandise through Port for Projects in MCC Compacts That Ended in 2010 and 2011

Thousand tons/year
Country                            Baseline                         Original target              Final target                 Final result
           a
Cape Verde                              482                                    711                        711                           N/C
Benin                                4,519b                                  6,945                      6,945                         7,606
                                        Source: GAO analysis of MCC data.

                                        N/C = not complete.
                                        a
                                         MCC eliminated performance indicators after decreasing the scope of the Cape Verde port project,
                                        because the indicators could not be measured prior to completion of the Phase II works, including the
                                        wharf 1, container yard, and breakwater. Phase II was funded by the government of Portugal and was
                                        expected to be completed after the compact closed.
                                        b
                                         Baseline from 2004. According to MCC, the value in 2005 was 5,680 thousand tons. If MCC had
                                        used 2005 data for its baseline, the volume of merchandise would not show as large an increase.



Table 12: Container Ship Time at Berth for Port Projects in MCC Compacts That Ended in 2010 and 2011

Days
Country                            Baseline                         Original target              Final target                 Final result
           a
Cape Verde                             1.41                                   1.01                       1.01                           N/C
Benin                                        2                                   1                           1                         1.31
                                        Source: GAO analysis of MCC data.

                                        N/C = not complete
                                        a
                                         MCC eliminated performance indicators after descoping the Cape Verde port project, because the
                                        indicators could not be measured prior to completion of the Phase II works, including the wharf 1,
                                        container yard, and breakwater. Phase II was funded by the government of Portugal and was
                                        expected to be completed after the compact closed.



Table 13: Container Ship Time at Anchor for Port Projects in MCC Compacts That Ended in 2010 and 2011

Hours
Country                            Baseline                         Original target              Final target                 Final result
           a
Cape Verde                                   5                                   4                           4                          N/C
Benin                                       16                                   4                           4                         34.6
                                        Source: GAO analysis of MCC data.

                                        N/C = not complete.
                                        a
                                         MCC eliminated performance indicators after descoping the Cape Verde Port project, because the
                                        indicators could not be measured prior to completion of the Phase II works, including the wharf 1,
                                        container yard, and breakwater. Phase II was funded by the government of Portugal and was
                                        expected to be completed after the compact closed.




                                        Page 39                                           GAO-12-631 MCC Transportation Infrastructure
Appendix V: Comments from the Millennium
             Appendix V: Comments from the Millennium
             Challenge Corporation



Challenge Corporation




             Page 40                                    GAO-12-631 MCC Transportation Infrastructure
Appendix V: Comments from the Millennium
Challenge Corporation




Page 41                                    GAO-12-631 MCC Transportation Infrastructure
Appendix V: Comments from the Millennium
Challenge Corporation




Page 42                                    GAO-12-631 MCC Transportation Infrastructure
Appendix VI: GAO Contacts and Staff
                  Appendix VI: GAO Contacts and Staff
                  Acknowledgments



Acknowledgments

                  David Gootnick, (202) 512-3149 or gootnickd@gao.gov
GAO Contact
                  In addition to the contact named above, Emil Friberg, Jr. (Assistant
Staff             Director), Miriam Carroll Fenton, Aryn Ehlow, Heather Hampton, and
Acknowledgments   Seyda Wentworth made key contributions to this report. In addition,
                  Michael Armes, Lynn Cothern, Reid Lowe, Ernie Jackson, Leslie Locke,
                  and Jena Sinkfield provided technical assistance.




                  Page 43                               GAO-12-631 MCC Transportation Infrastructure
Related GAO Products
             Related GAO Products




             Millennium Challenge Corporation: Georgian and Benin Transportation
             Infrastructure Projects Varied in Quality and May Not Be Sustainable.
             GAO-12-630. Washington, D.C.: June 27, 2012.

             Millennium Challenge Corporation: Compacts in Cape Verde and
             Honduras Achieved Reduced Targets. GAO-11-728. Washington, D.C.:
             July 25, 2011.

             Millennium Challenge Corporation: Summary Fact Sheet for 17
             Compacts. GAO-10-797R. Washington, D.C.: July 14, 2010.

             Millennium Challenge Corporation: MCC Has Addressed a Number of
             Implementation Challenges, but Needs to Improve Financial Controls and
             Infrastructure Planning. GAO-10-52. Washington, D.C.:
             November 6, 2009.

             Millennium Challenge Corporation: Independent Reviews and Consistent
             Approaches Will Strengthen Projections of Program Impact. GAO-08-730.
             Washington, D.C.: June 17, 2008.

             Management Letter: Recommendations for Improvements to MCC’s
             Internal Controls and Policies on Premium Class Air Travel.
             GAO-08-468R. Washington, D.C.: February 29, 2008.

             Millennium Challenge Corporation: Projected Impact of Vanuatu Compact
             Is Overstated. GAO-07-1122T. Washington, D.C.: July 26, 2007.

             Millennium Challenge Corporation: Vanuatu Compact Overstates
             Projected Program Impact. GAO-07-909. Washington, D.C.:
             July 11, 2007.

             Millennium Challenge Corporation: Progress and Challenges with
             Compacts in Africa. GAO-07-1049T. Washington, D.C.: June 28, 2007.

             Millennium Challenge Corporation: Compact Implementation Structures
             Are Being Established; Framework for Measuring Results Needs
             Improvement. GAO-06-805. Washington, D.C.: July 28, 2006.

             Analysis of Future Millennium Challenge Corporation Obligations.
             GAO-06-466R. Washington, D.C.: February 21, 2006.




             Page 44                            GAO-12-631 MCC Transportation Infrastructure
           Related GAO Products




           Millennium Challenge Corporation: Progress Made on Key Challenges in
           First Year of Operations. GAO-05-625T. Washington, D.C.:
           April 27, 2005.

           Millennium Challenge Corporation: Progress Made on Key Challenges in
           First Year of Operations. GAO-05-455T. Washington, D.C.:
           April 26, 2005.




(320801)
           Page 45                           GAO-12-631 MCC Transportation Infrastructure
GAO’s Mission         The Government Accountability Office, the audit, evaluation, and
                      investigative arm of Congress, exists to support Congress in meeting its
                      constitutional responsibilities and to help improve the performance and
                      accountability of the federal government for the American people. GAO
                      examines the use of public funds; evaluates federal programs and
                      policies; and provides analyses, recommendations, and other assistance
                      to help Congress make informed oversight, policy, and funding decisions.
                      GAO’s commitment to good government is reflected in its core values of
                      accountability, integrity, and reliability.

                      The fastest and easiest way to obtain copies of GAO documents at no
Obtaining Copies of   cost is through GAO’s website (www.gao.gov). Each weekday afternoon,
GAO Reports and       GAO posts on its website newly released reports, testimony, and
                      correspondence. To have GAO e-mail you a list of newly posted products,
Testimony             go to www.gao.gov and select “E-mail Updates.”

Order by Phone        The price of each GAO publication reflects GAO’s actual cost of
                      production and distribution and depends on the number of pages in the
                      publication and whether the publication is printed in color or black and
                      white. Pricing and ordering information is posted on GAO’s website,
                      http://www.gao.gov/ordering.htm.
                      Place orders by calling (202) 512-6000, toll free (866) 801-7077, or
                      TDD (202) 512-2537.
                      Orders may be paid for using American Express, Discover Card,
                      MasterCard, Visa, check, or money order. Call for additional information.
                      Connect with GAO on Facebook, Flickr, Twitter, and YouTube.
Connect with GAO      Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts.
                      Visit GAO on the web at www.gao.gov.
                      Contact:
To Report Fraud,
Waste, and Abuse in   Website: www.gao.gov/fraudnet/fraudnet.htm
                      E-mail: fraudnet@gao.gov
Federal Programs      Automated answering system: (800) 424-5454 or (202) 512-7470

                      Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-
Congressional         4400, U.S. Government Accountability Office, 441 G Street NW, Room
Relations             7125, Washington, DC 20548

                      Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800
Public Affairs        U.S. Government Accountability Office, 441 G Street NW, Room 7149
                      Washington, DC 20548




                        Please Print on Recycled Paper.