Economic Development: Observations Regarding the Economic Development Administration's May 1998 Final Report on Its Public Works Program

Published by the Government Accountability Office on 1999-03-23.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

      United States
GAO   General Accounting
                    D.C. 20548

      Resources, Community,        and
      Economic Development         Division


      March 23,1999

      The Honorable John R. Kasich
      Chairman, Committee on the Budget
      House of Representatives

      Subject:       Economic Develonment: Observations Regarding the Economic Development
                     Administration’s Mav 1998 Final Renort on Its Public Works Program

      Dear Mr. Chainnan:

      You requested that we provide you with our observations on the analyses presented in the
      Economic Development Administration’s (EDA) May 1998 report entitled Public Works
      Program-Multinlier and Emnlovment-Generating Effects. Final Renort. The report
      concludes that EDA’s public works program “does indeed produce permanent private-
      sector employment at a relatively low cost.” We agreed to examine the report’s regression
      analyses’ to determine whether the report’s results would be similar if additional variables
      that directly took into account the prior level of a county’s employment or population
      were included in the analyses. We also agreed to determine whether this report’s job
      creation estimates include only new jobs or might also include jobs relocated from another
      area and to examine other technical issues that might affect the reliability of the cost
      estimates presented.

      As we stated in our 1996 report, we believe that attempting to quantify the gains from
      economic development programs is difficult.2 A persuasive study of the impact of a
      development program would have to demonstrate an improvement in the economy of an
      area receiving assistance, link that improvement to an agency’s programs, and rule out
      alternative causes. In our 1996 report, we found no studies that met these criteria. In part
      because of our report, EDA commissioned a series of studies-including the May 1998
      report-on its public works program.

      ‘Regression analysis is a technique used to estimate the kfluence of individual factors while holding constant,
      or controlling for, the effects of other factors. The analyses in the EDA report estimated the innuence of EDA
      grants on employment while controlling for the effects of other factors that influence employment, such as
      whether a county receiving a grant is in a metropolitan area.

      2Economic Development: Limited Information Ekists on the ImDact of Assistance Provided bv Three Agencies
      (GAO/RCED-96-103,Apr. 3,1996).

                                                               GAO/RCED-99-11R EDA’s May 1998 Final Report


We do not believe that the conclusion in the EDA report about the linkage between its
public works program and increases in permanent private sector jobs is warranted. Our
analyses produced different results from the EDA study. When we included each county’s
level of employment prior to the EDA grant in our regression analyses, we found that prior
employment levels had a significant effect on counties’ post-grant employment levels, and
EDA grants no longer had a significant effect. By not taking into account the counties’
previous employment levels, the study may have shown only that larger counties-with
high levels of employment-tend to receive the most grants. In addition, the job creation
estimates did not address whether the jobs are new or relocated from another area but
only that counties receiving EDA grants have more jobs than counties not receiving grants.
  We also identified some technical issues pertaining to the certainty of the cost estimate
per job created. For example, the report’s conclusion that an additional $10,000 in EDA
 spending in a typical county would yield nine new jobs is applicable only if that county
were receiving a grant substantially smaller than the average grant EDA made.


EDA was established in 1965 within the Department of Commerce to target federal
resources to economically distressed rural and urban communities across the country that
lagged in economic development, industrial growth, and personal income. EDA’s budget
authority for fiscal year 1998 was $340 million, and its obligations for its public works
grants were $178 million. The public works and development facilities program, EDA’s
program with the largest amount of funding, provides grants to help distressed
communities attract new industry, encourage business expansion, diversify economies,
and generate long-term private sector jobs. Selection criteria for public works grantees
favor distressed areas with high unemployment; however, these areas may be within
 counties that do not have overall levels of distress or high unemployment.

EDA commissioned two studies on the effect of its public works funding on job creation.
The first study, entitled Public Works Program: Performance Evaluation, Final Renort
(May 1997), used a combination of surveys, seminars for grantees, and site visits to obtain
performance information from 203 EDA projects that received their final payment in 1990.
 The study found that 96 percent of the projects produced permanent jobs 6 years after
completion at a cost to EDA of $3,058 per job created or retained. The second study,
entitled Public Works Program-Multinlier and Emnlovment-Generating Effects. Final
Renort (May 1998), employed two techniques to document the effects of EDA’s public
works projects on the employment growth of counties receiving grants. The study builds
on the findings from the first study. Using the first technique, researchers found that for
every 10 jobs that can be directly attributed to an EDA project, an additional 5 jobs will be
created or moved to the county in which the project is placed. The second technique,
regression analysis, showed that $10,000 increments in public works grant spending
produce, on average, nine jobs, for an estimated cost of $1,100 per job created or moved to
a county that receives an EDA grant.

 2                                            GAO/RCED-99-IlR   EDA’s May 1998 Final Report


We believe that the results of EDA’s study are highly dependent on the model specification
used and that the study’s definitive conclusions are unwarranted. By changing the
specification3 of the model to control for the effect of prior county employment, we
obtained substantially different results that show that EDA expenditures did not have a
significant effect. To fully evaluate the impact of EDA grants on employment, it is
important to consider the impact using the results from a broader range of specifications.

EDA’s study used county wages, house prices, urban status, the recent performance of the
labor market (for example, the growth of employment from 1988 through 1990), and the
demographic characteristics of the population as factors to take into account in estimating
the effects of EDA grants.’ Taking these factors into account, the researchers found that
EDA’s expenditures have a statistically significant, positive effect on county employment
levels. However, we do not believe that the study sufficiently accounted for county-level
effects, such as the prior levels of employment and the population, that could also be
associated with current employment levels.

Taking county-level effects into account is important because EDA’s spending in 1990 was
associated with a county’s prior levels of employment and population- Without accounting
for these effects, EDA’s study may be showing only that larger counties with higher levels
of employment tend to receive the most grants-rather than measuring the impact of EDA’s
grants on employment growth. When we examined the placement of the 203 EDA projects
in this study, we found that the average county receiving a grant had a 1986 employment
level of about 58,000 and a 1985 population of about 162,000. However, the average county
not receiving a grant had an average 1986 employment level of about 24,600, and a 1985
population of about 70,500. Examinin g the most populous county in each state, we found
that 24 percent received one (or more) public works grants; 6 percent that were neither
the largest nor the smallest in population received one or more grants; and only 2 percent
of the counties with the smallest population in each state received a grant.

We conducted several analyses using additional variables-such as county employment and
population-that were not included in EDA’s study. These analyses enabled us to assess
whether EDA grants would still have a significant effect in regression analyses that took
into account the effects of these additional factors on employment. For example, when
we used employment levels in 1986 as a factor in our regression analyses, the analyses
showed that EDA’s expenditures in 1990 had no significant effect on employment levels
for 1990 through 1994 in the counties receiving grants. Including the county’s population
in 1985 instead of county employment as a factor, we also found that EDA’s expenditures

%y model specification we mean the specific set of variables included in the model and the specific
assumptions made about how those variables are related to each other.

‘In addition to these factors, the researchers took into account factors particular to the state economy, which
are sometimes called state fixed-effect variables.

3                                                       GAOLRCED-99-11R EDA’s May 1998 Final Report

had no effect on employment levels. We chose to use 1985 population and March 1986
employment data because most of the projects in the analyses were approved in 1985 or
later. Therefore, it is unlikely that the projects had any measurable impact on population
or employment in March 1986 or before.5 At the suggestion of the EDA contractors, we
also controlled for 1988 employment. Again, we found that EDA expenditures had no
significant effect on employment levels. (See enc. I for the detailed specifications and
results of our analyses.)


EDA’s regression analyses did not address whether the jobs attributed to EDA grants are
new or relocated. The analyses attempted to demonstrate that, holding constant other
influences, such as whether the county is in a metropolitan area, counties that receive
EDA grants had more jobs than counties that did not. As a result, the job impact in a
particular county that is measured in the study may include newly created jobs in that
county, jobs that moved to the county from another location, or jobs that would have
moved to another county but were retained because of the grant. Because the analyses did
not measure whether these jobs are new or simply relocated, some statements in the
report overstate the regression results. For instance, on page 43, the report states, “EDA
public works investments do, in fact, create jobs,” and on page 45, it states, “the EDA
Public Works Program is a significant (in both a statistical and numerical sense)
 contributor to the productive capacity of the country.” While the regulations for the public
works program prohibit the use of grants to relocate jobs from one area to another, EDA
 relies on assurances from applicants and certifications from businesses benefiting from
 the assistance that the businesses will not make such transfers. However, these
 assurances do not account for market impacts. Even if a county receiving an EDA grant
 has additional jobs because of that grant, the economy as a whole may not have
 necessarily gained jobs. For example, if an EDA grant leads to increased jobs in a
 particular industry in one county, then the increased output in that industry may lead to
 lower prices, which, in turn, may result in reduced output and employment elsewhere.


We identified two additional technical issues that could affect the estimates of the cost per
job presented in EDA’s 1998 report. First, to use their model specification, EDA’s
 contractors needed to make an arbitrary decision on how to include the key predictor
variable-the level of EDA’s expenditures-for the large majority of counties that did not
 receive grants. We found that by using different arbitrary adjustments of that variable but
 otherwise following the method used in EDA’s report, we obtained different estimates-
 both higher and lower-of the cost per job created.” For example, over the range of

 ‘According to EDA’s contractors, about 5 percent of the projects were completed before 1986.

 “In their andyses, EDA’s contractors used a specilkation in which they used the natural logarithm of the

 4                                                      GAO/RCED-99-11R EDA’s May 1998 Find Report
’ B-281059

  ahernatives we used, we obtained estimates of the cost per job created ranging from about
  $600 to about $1,600, compared with the estimate of $1,100 in EDA’s report.         -

  Second, the report’s conclusion that an additional $10,000 in EDA spending in a typical
  county would yield nine new jobs is applicable only if that county were receiving a grant
  substantialIy smaIIer than the average grant EDA made. The model specification that
  EDA’s contractors used implies that the number of new jobs resulting from a specified
  increase in EDA spending (such as $10,000) is different at different levels of spending.7
  EDA’s contractors estimated the effect of a $10,000 spending increase when the grant level
  was about $46,500.’ However, the average grant during this period was about $750,000.
  We found that the estimates in EDA’s report implied that a $10,000 increase in EDA
  spending in a county receiving an average grant, and with average employment among
  counties receiving grants, would yield about six new jobs, rather than the nine new jobs
  that the EDA report concluded would be created. This finding of six new jobs would raise
  the estimated cost per job created from about $1,100 to about $1,600.’


  We provided a draft of this report to the Department of Commerce for review and
  comment. The Department’s Economic Development Administration (EDA) provided
  comments on the report. EDA disagreed with our findings regarding the effect of its

  variables, rather than the variables themselves, in the regression analyses. The natural logarithm of a number
  is the power to which e (approximately 2.7) must be raised to produce that number. When they created the
  logarithm for EDA’s expenditures, they added a value of 1 x 10 p only to the counties that did not receive a
  grant (and therefore had a value of zero), in effect treating those counties as if they received grants equal to a   .
  small fraction of one cent. Some adjustment was necessary because the logarithm of zero is undefined
  However, the choice of 1 x 10” was arbitrary, and when we used adjustments ranging from 1 x 10 .’ to 1 x lo’@
  instead, we obtained estimates of the cost per job ranging from a little more than 50 percent of the estimate in
  EDA’s report to nearly 50 percent more than the estimate in the report. An alternative approach to dealing
  with this situation is to add 1 to the level of EDA expenditure for each observation before creating the

  ?.f the contractors had used a linear specification, then the estimated change in employment for a $10,000
  grant increase would be the same regardless of the initial grant level. However, in a logarithmic specification
  such as the one used in EDA’s report, the estimated percentage change in employment would be the same for
  any given percentage change in EDA spending but the absolute change for a given absolute change in spending
  would vary.

  8The contractors chose this level because it represented the mean grant value in the data set used for
  estimation, which includes counties not receiving grants (total EDA grant spending divided by the number of
  counties). Because most counties did not receive grants during this period, this number is substantially lower
  than the average grant EDA made.

  ‘When the contractors estimated the number of new jobs created from a $10,000increase in EDA spending,
  they did not use a measure of employment that appropriately corresponded to the $46,500measure they used
  for grant size. If we had used EDA’s method of selecting an employment level to correspond with a grant level,
  we would have found that the estimates in EDA’s report imply that a $10,000increase to a $750,000grant
  would create about 1.5 new jobs at a cost per job created of about $6,700.

  5                                                         GAOLRCED-99-1lR EDA’s May 1998 Final Report

investments on county employment. EDA stated that our findings were based on an
incomplete statistical model. It said that if our economic model took into account prior
compensation levels for jobs in each county as well as prior employment, its grants would
be shown to have a positive effect on employment. EDA’s comments mischaracterized our
findings. It was not our objective to create another model but r&her to evaluate the
analyses and results contained in EDA’s May 1998 final report on the public works
program. We did not take into account prior compensation levels for jobs in each county
in our analyses because EDA’s May 1998 report showed few effects attributable to
 compensation levels. After reviewing EDA’s comments, we investigated a number of
model specifications that took into account prior compensation levels as well as prior
 employment. We found that under certain assumptions EDA grants had significant effects
 on employment while in others they did not. Whether or not EDA grants had a significant
 effect depended on which prior year’s data were used and how growth in wages and
 employment were modeled. This inconsistency confirms our view that the results of
 EDA’s study were highly dependent on the model specification used and that the study’s
 definitive conclusions were unwarranted. Therefore, we made no changes to the report.

EDA also disagreed with the report’s discussion on job creation. EDA said that its grants
create new jobs in highly distressed local economies and that, because of legislative and
regulatory restrictions, new jobs resulting from job relocations are unlikely. EDA’s
comments on the relocation issue cover only part of our concern about the job creation
estimates in EDA’s report. Our main concern is that EDA’s analyses do not address
whether the jobs attributed to its grants come at the expense of jobs elsewhere in the
economy. Therefore, some statements in the EDA report indicating that EDA investments
do create jobs overstate the results of the regression analyses. We made no changes to the
report in response to EDA’s comments. EDA’s comments and our detailed responses
appear in enclosure II.


In order to examine the regression analyses in EDA’s report, we obtained the data set used
by EDA’s contractors. We then recreated some of the regression results of EDA’s
contractors to confirm our understanding of the data set. Next, we added employment
data from the 1986 and 1988 Countv Business Patterns and 1985 population data from the
Bureau of the Census. We used these additional data to test alternative specifications to
examine whether similar findings regarding the association between the level of EDA
grants in a county and that county’s employment level would be obtained using these
 alternatives. A detailed description of our methodology appears in enclosure I.

To examine other technical issues that might affect the reliability of the cost estimates, we
reviewed the report’s methodology. As we identified areas of concern, we assessed their
implications. We also assessed the methodology to determine if it could be used to
identify whether jobs associated with EDA grants were necessarily jobs that had been
newly created.

 6                                            GAO/RCED-99-1lR EDA’s May 1998 Find Report

We are sending copies of this report to Senator Robert C. Byrd, Senator Joseph I.
Lieberman Senator Ted Stevens, and Senator Fred Thompson and to Representative Dan
Burton, Representative David R. Obey, Representative Henry A. Waxman, and
Representative C.W. Bill Young in their capacities as Chair or Ranking Minority Member of
Senate and House Committees. We are also sending copies of this report to The
Honorable William M. Daley, the Secretary of Commerce. Copies will also be made
available to others on request.

If you or your staff have any questions about this report, please call me on (202) 512-7631.
Major contributors to this report were Susan Campbell, Jay Cherlow, and Austin Kelly.

Sincerely yours,

Judy A. England-Joseph
Director, Housing and Community
 Development Issues

Enclosures - 2

                                              GAO/RCED-99-11R EDA’s May 1998 Final Report
ENCLOSURE I                                                                                  ENCLOSURE I

                                     RESULTS OF OUR ANALYSIS

This enclosure describes the alternative model specifications we tested, the variables we
used, and the results of several regressions we ran. Our purpose was to examine whether
the Economic Development Administration’s (EDA) conclusion; that a statistically
significant association exists, between the level of EDA grants in a county and that
county’s employment level, would be reached using alternative model spectications. In
particular, we tested whether this statistically significant association remained once we
controlled for the county’s level of employment at an earlier date or for the county’s
population. We found that in alI of the specifications we used, the relationship between
EDA grants a county received and the employment level in that county was not
statistically different from zero. That is, these results suggest that (1) the report’s
findings are highly sensitive to the specification of the model used and (2) the signiscant
relationship found when the prior county employment level is not taken into account may
reflect the fact that the highest levels of EDA grants go to the counties with the highest
 employment levels rather than that there is any causal relationship between EDA grants
 and employment.


EDA’s contractors used a database that included the total grant amount for EDA projects
receiving final funding in 1990 for each county in the United States. They used this
database to run several speciiications of a two-equation, pooled, cross-section, time-
series model; that is, observations on each county for each of the 5 years were pooled, or
combined. The dependent variables were the natural logarithms of the levels of
employment and payroll per worker (in 1990 dollars).

We tested several specifications that were modifications of the report’s “specification 3.”
These specifications, which employ the two-stage least squares (2SLS)’ regression
procedure used in specilication 3 of the report, were ail designed to test whether a
county’s level of EDA grants continue to show a statistically significant association with
the county’s employment level once we control for the county’s prior level of employment
or population.’ We believe that by omitting this factor, the report’s results may overstate
the importance of EDA grants on a county’s employment level because of the association
between EDA grant spending in 1990 and prior levels of employment and population.

‘2SLSis an econometric technique used when a model contains two or more dependent variables, in this case,
employment and pay, that simultaneously infhrence each other and are thereby jointly determined.

 2Weused specification 3 as the basis for our alternatives for two reasons: (1) the results from specification 3
 are the ones EDA uses to estimate the cost per job and (2) specification 3 is the only alternative presented
 that considers a county’s compensation level as a factor explaining employment level, although the report
 says that “clearly, a well-specified model of employment must include compensation.” In subsequent
 correspondence, EDA’s researchers have suggested that despite what the report says, regression using
 reduced form ordinary least squares @IS) rather than 2SLSmay be more appropriate. Unlike 2SIS, OIS
 regression does not treat the dependent variables, compensation and employment, as being jointly
 determined. Since we were asked to review the analyses used in the report as the basis for the conclusion
 about the impact of EDA grants, and the report makes a strong casefor using 2SI.Sbecause of the complex
 relationship between compensation and employment levels, we used 2SLSin our specifications. Moreover,
 even if regressions using OLS that control for prior employment or population levels found that the level of
 EDA grants remained a statisticaIly significant factor, the results from 2SLSregressions would remain
  important in assessingthe impact of EDA grants.

 8                                                         GAOIRCED-99-11R EDA’s May 1998 Final Report
ENCLOSURE I                                                                                ENCLOSURE I

We used several specifications to estimate the significance of EDA grants while
controlling for prior employment or population levels.3 Because we wanted to measure
employment before the grants could have had an impact, we used the employment levels
in 1986 as a control variable. We conducted two regression analyses including the natural
logarithm of that variable, one that included all the variables in the report’s specification
3 and one that omitted the variable measuring growth in employment from 1988 to 1990.
Although using a‘later year would seem to run a greater risk that some of the grant
projects might already have been completed and begun having an impact, at the
suggestion of EDA’s contractors, we also used a specification that used the employment
level in 1988. In that specification, we omitted the growth in employment from 1988 to
 1990 because the two variables together would equal 1990 employment. Finally, we also
used a specification in which we used the natural logarithm of 1985 population as a
control variable. Population is a good, although not perfect proxy for employment, and is
almost certainly not substantially affected by the grants themselves.

3Weestimated a regression that was identical to specification 3 to confirm that we understood the data set
well enough to replicate EDA’s contractors’ results. Our coefficient estimates were virtually identical to
those in the EDA report.

                                                       GAOLRCED-99-11R EDA’s May 1998 Final Report
ENCLOSURE I                                                                              ENCLOSURE I


Table I.1 summarizes the dependent and independent-or predictor-variables                   that were
used in the analysis along with their corresponding means.

Table 1.1: Definitions and Mean Values of Variables Used in the Analvsis

Variable                          1 Definition                         1 Mean
Detsendent variable

LTEMPMM”                           The natural logarithm of                                           8.68
                                   county employment in the
                                   relevant year,. 1990-l 994
Predictor variable

SMSHARE                           1The proportion of the                                             79.32
                                    county’& firms with fewer
                                    than 10 employees
BIGSHARE                            The proportion of the                                             0.04
                                    county’s firms that have
                                    more than 1,000 employees
URBAN                               Dummy variable equals 1 if                                        0.27
                                    a county is in a
                                    Metropolitan Statistical
                                    Area (MSA); otherwise it is
LHVSO                               The natural logarithm of                                         10.79
                                   - 1990 county house value
GREM8890                            The percentage growth in                                           0.03
                                  ( county employment from        1                                           I
                                     1988 to 1990
GRWA8890                             The percentage growth in                                         -0.02
                                     payroll per county worker
                                     from 1988 to 1990
LPAY                                 The natural logarithm of                                          9.70
                                     county payroll per worker
LEM P86.                             The natural logarithm of                                          8.53
                                      1986 county employment
LEMP88                               The natural logarithm of                                          8.64
                                      1988 county employment
 LPOP85                              The natural logarithm of                                         10.14
                                      1985 county population
 LEDA                                The natural logarithm of the                                    -18.69
                                      county’s EDA grant value in
                                   , ;:9Tlevant year, 1990-       ,

 “The natural logarithm of employment was used as a predictor variable in the model’s other equation that
 had county payroll per worker as the dependent variable.

 10                                                    GAOIRCED-99-11R EDA’s May 1998 Final Report
ENCLOSURE I                                                                                     ENCLOSURE I


Table I.2 shows the second-stage regression results for EDA’s specification 3 with the
logarithm of employment as the dependent variable and the four alternative
specifications we tested.4 In each case, the coeffkient on our measure of prior
employment or population level is highly significant, while the coefficient on our measure
of the value of EDA’s grants is not statisticaZly different-or even close-from zero at
conventional levels.

Table 1.2: Reqression Results for EDA’s Specification 3 and Our Four Alternative
SDecifications    .
                            Coefficient estimates and standard errors’
Predictor               EDA’s        GAO’s        GAO’s         GAO’s                            GAO’s
variable                Spec. 3      Alt. 1       Alt. 2        Ah. 3                            Alt. 4

1 SMSHARE           I       -0.112b 1          -0.025b 1          -0.025b 1         -O.OOgb1              -0.093b 1
                            (0.008)           (0.003)             (0.003)           (0.001)               (0.011)
 BIGSHARE                    0.941 b            0.825b             0.908b            0.347b                2.983b
                           (0.235)            (0.148)             (0.184)           (0.063)               (0.521)
 URBAN                       o.454b             0.068b             0.070b            0.021 b              -0.141b
                           (0.028)            (0.011)             (0.012)           (0.004)               (0.033)
 LHVSO                       0.962b             0.428b             0.44gb            0.1 32b               1 .252b
                            (0.107)          (0.0522)             (0.062)           (0.021)               (0.190)
 GREM8890                    0.266b            -0.064b                 ---                ---                   w--
                           (0.015)             (0.020)                                                                   .
 GRWA8890                   -0.835b             0.951 b             0.842b            0.248b            2.590b
                           (0.305)            (0.248)             (0.229)           (0.077)            (0.605)
 LPAY                        1 .422b            ‘-2.63b            -2.762b           -0.764b           -7.434b
                            (0.775)            (0.621)            (0.703)            (0.232)           (1.797)
 LEDA                      0.0074b             0.0001           -0.00005          -0.00009            -0.0016
                          (0.0007)           (0.0005)            (0.0005)          (0.0002)          (0.0013)
 LEMP86                           --i            1 .054b          1 .0003b                ---               ---
                                               (0.034)             (0.019)
 LEMP88                          m-e                  ---               ---          0.99tsb                   ---
 LPOP85                          ---                 -we                 m-m             ---               1 .207b
Summary statistics
Observations       15,591                      15,552              15,552           15,592                15,587
Adjusted           0.8545                       0.944               0.937           .0.991                 0.665

‘All regressions include year and state fixed effects, the results of which are available upon request.

 %tatistically different from zero at the 99 percent confidence level.

 4~though our emphasis was on the effect of EDA gi-ants on county employment, because we were using
 modifications of the contractor’s 2SIS estimation procedure, we also estimated specification 3’s payrou
 equation with similar modifications. We have not reported those results here, but they are available on

 11                                                        GAO/RCED-99-1lR     EDA’s May 1998 Final Report
            ENCLOSURE II                                                                                    ENCLOSURE II


Note: GAO
comments supplementing
those in the report
appear at the end
of this enclosure

                                                                                THE SECRETARY             OF COMMERCE
                                                                                Washington.   OX. 20230
                                                                                  FEE I 2 I%cl

              Ms. Judy A. England-Joseph
              Housing and Community
               Development Issues
              U.S. General Accounting Office
              Washington, DC 20548

              Dear Ms. England-Joseph:

              Thank you for your letter requesting the Department’s comments on the draft General
              Accounting Office (GAO) report entitled “Economic Development: Observations Regarding the
              Economic Development Administration’s (EDA) May 1998 Final Report on its Public Works
              Program.” We appreciate the opportunity to provide comments, which support the results of the
              independent evaluation of the EDA Public Works program.

              EDA management and professional staff and members of the research team have reviewed the
              report, and worked with GAO to review the methodologies used in the evaluation. The EDA-
              commissioned research project represents management’s strong commitment to program
              evaluation. Comprehensive comments from the Assistant Secretary for Economic Development
              and the Rutgers University research team are attached.

              We appreciate your interest in the Department and its programs.


             12                                                    GAOLRCED-99-1lR EDA’s May 1998 Final Report
      ENCLOSURE II                                                                                       ENCLOSURE II

                                                   UNITED      STATES       DEPARTMENTOF                COMMERCE
                                                   Thu Assistant       Sam-etary     for Economic         Dsvalopment
                                                   Washragton.     DC.    20230

           MEMORANDUM FOR               THE SECRETARY                            --I     f-   .
           FRO-M:                       Phillip A. Singennan,Ph.D( -&a         ‘( j&,.;c.Aw--
                                        Assistant Secretary      ; -                                :
                                        for EconomicDevelopment

           SUBJECT:                     GAO Review of EDA’s May 1998Final Report on its Public
                                        Works Program

           We appreciate the opportunity to respondto the GAO review of EDA’s Public Works program
           evaluation. Over the past six months EDA, GAO, and EDA’s researchteams have worked
           togetherto review the methodoIogiesusedin the evaluations. In addition, we take this
           oppommity to thank the GAO staff for their collaboration with us on our program evaluation
           efforts and on our commitment to fully comply with the requirementsof the Government
           Performanceand Results Act of 1993.

           Two researchteams,both led by Dr. Robert W. Burchell of Rutgers University, conducted
           studiesof EDA’s Public.Works program resulting in the 1997and 1998reports. In this work,
           they undertookground-breaking researchin evaluating the impact of EDA’s major program.
           Their evaluations,using several independentmethodologies,have confirmed the effectiveness
           of EDA’s investmentsin creating permanentjobs. The researchconfirmed that EDA projecis
           createsubstantialnumbersof jobs and do so in areasof significant distress,areaswhere the
           averageunemploymentlevels, percent of the population in poverty, and per capita income were
           40 percent worse than stateand national averages..Furthermore,every Sl million of EDA
           riding leveraged 510.08 million in private sectorinvestmentand increasedthe local tax base
           byS10.13 million.

           However, the estimatedaverage cost of thesejobs varies slightly between the two Rutgers
           t,:ams’ methodologiesand between Rutgersand GAO. Basedon a detailed analysis of 100
           percent of the Public Works projects completedin FY 1990-more than 200 projects-the first
           researchteam conlirmed that EDA projectscreateddirectjobs at an average cost of about
           S3,OOO.The secondresearchteam found slightly lower job-cost estimatesusing econometric
           techniques. EDA’s job creation costsvary dependingon the size and income levels of an area:
           in larger and poorer counties the costsaremuch lower than in smaller countieswhere wage rates
           are higher.

           Both the se&d Rutgers team and GAO employ standard,academically acceptedstatistical
           methodologies. However, the methodsavailable to the.researcherare less than totally precise
           tools, depending as they do on the assumptionsand variables they employ. As such, within the
           academiccommunity there is some room to disagree. After careful consideration and following
           consultation with the Rutgers researchteam that conductedthe econometricstudies,we must

      13                                                      GAOIRCED-99-1lR EPA’s May 1998 Final Report

..   -..
            ENCLOSURE II
                                                                                                     ENCLOSURE II

                 respectfully and firmly disagreewith GAO’s observations regarding the effect of EDA
                 investments on county employment. We believe conclusions drawn by GAO are based on an
See Comment 1.   incomplete statistical model used in their research. In fact, if properly developed, the data
                 support the conciusion that the effect of EDA Public Works investments on county employment
                 is p&rive and sraristicaI[v significunf at high levels of confidence. The research team’s review
                 of the GAO methodology is set forth in the Attachment.

                 EDA funding createsjobs in highly distressed local economies. These indeed are new jobs, not
                 ones relocated from elsewhere. The high level of economic distress in assisted areas, EDA’s
See comment 2.   legislative exhortation against funding job relocations, as well as EDA’s internal implementing
                 procedures make funding job relocations most unlikely. Furthermore, research undertaken by
                 economists such as David Birch and others, has shown that the number of jobs in any region
                 created by the relocation of industries or firms is quite insignificant.

                 The Rutgers researchteam’s review of the underlying methodology is attached. EDA fully
                 endorses its methodologies and findings.


           14                                                  GAO/RCED-99-UR         EDA’s May 1998 Final Report
        ENCLOSURE II                                                                               ENCLOSURE II

                                             RUTGERS RESEARCH TEAM RESPONSE
                                         TO GAO’S “OBSERVATIONS REGARDING THE
                                        ECONOMIC DEVELOPMENT ADMINISTRATION’S
                                   MAY 1998 FINAL REPORT ON ITS PUBLIC WORKS PROGRAM”

                               The research team appreciates GAO’S diligence in pursuing this important question, its
                       willingness to provide us with the details of its analyses, and the opportunity to respond in this
See coqment 3.         document. The research ‘team respectfully disagrees, however, with the conclusions GAO draws
                       regarding the effect of EDA investments on county employment. The conclusions drawn by
                       GAO are based on an incomplete statistical model, as will be explained below. First, some
                       background information on the various EDA studies and their results.

                       Background on the EDA Public Works Reports
                               Two studies and reports were completed on the impacts of EDA funding on job creation:
                                The first report, Public Works Program: Performance Evaluation (May 1997), found that
                       96 percent of 203 EDA projects produced permanent employment in the location of their
                       development at a cost of $3,058 per direct job. This information was received from the field
                       through both telephone interviews and responses to a written information request. One hundred
                       percent of the projects were contacted and provided information. Thirty percent of the sites (60
                       projects) were actually visited by research team members, who verified fast-hand the
                       information reported by the survey. Upon visiting the sites, research team members entered
                       buildings and counted jobs. Further, they interviewed the owners of the facilities and the EDA
                       grantees at the site. Btith business owners and EDA grantees agreed that these were not locations
                       where jobs would grow spontaneously-levels of unemployment and families living below the
                       poverty line were 40percent worse than state and national averages. EDA$nding was reported
                       as “critical” in creating new jobs in these areas.
                                The second report, Public Works Program: Multiplier and Emplo-vment-Generating
                       Eficts (May 1998), sought to determine whether EDA projects had an effect on job growth in
                        counties where this funding was present. The 203 public w.orks projects in 172 counties were
                        viewed in the context of jobs being created in 3,200 counties nationally. Two econometric
                        techniques were used: hput-output analysis found that EDA-funded direct permanent empioy-
                       ment produced indirect and induced permanent employment at a ratio of 2 to 1; that is, the
                        employment multiplier is 1.50. (Direct private-sector project investments similarly leverage
                        indirect and induced private investment under a comparable investment multiplier of 1.44.) The
                        regression analysis found that EDA funding had a statistically significant and positive effect on
                        county total employment levels. Further, counting aNjob growth (total jobs), EDA funding led to
                       jobs created at a cost of about $ 1, 115 per total job. This is roughly comparable to the finding that
                        EDA created direct jobs at a cost of $3,058 per direct job, since the total nlunber of jobs related
                       to EDA funding is likely to be 1.5 times the number of direct jobs. Dividing $3,058 by 1.5
                       indicates a total job cost of about $2,000 per job.
                              One source of data (regression analysis) revealed that EDA created jobs at a cost of
                       approximately $I,1 15 per job; a separate independent source of data (survey and physical
                       counting) confirmed that EDA created jobs at a cost of about $2,000 per total job. The
                       procedures employed in the May I998 regression analysis are standard econometric tests widely

        15                                                    GAO/NED-99-11R         EDA’s May 1998 Final Report
        ENCLOSURE II                                                                            ENCLOSURE II

                       employed in academic and practitioner circles for complex cause and effect relationships of the
                       kind evaluated here. They are sophisticated and complete. yielding results with which the
   See comment 4.      research team is comfortable. Further, their results basically paraUe1 findings from the Ma)
                       I997 stud\!, wherein these new jobs were actually counted. The fact that tx’o vety dtfferent
                       methodologies generated very similar job costfigures is important and noteworthy.

                       The GAO Statement of Facts, “Observations on EDA’s May 1998 Report”
                         i     GAO claims that the relationship between EDA public works funding and county
                       employment levels is statistically insignificant. This point is crucial, as it goes to the heart of the
                       matter: do EDA investments have a discernible effect on county employment? The researchteam
                       is convinced that the GAO conclusion is incorrect and that there is a positive relationship
                       between EDA investments and county employment, as stated in the May 1998 report.
                               The GAO analysis begins with the observation that EDA public works grants are not
                       randomly distributed across counties, and that failure to control for the unique characteristics of
                       these counties leads to a false positive relationship between EDA funds and county employment.
See comment 5.         GAO’s analysis emphasizes the fact that EDA funds are provided to counties that are, on
                       average, larger (in terms of their employment and population levels) than typical U.S. counties.
                       GAO argues that once this fact is acknowledged and controlled for with statistical techniques,
                       there is no relationship between EDA funds and county employment levels. The GAO conclusion
                       does not stand up to scrutiny, however. Three considerations lead the research team to the :
                       conclusion that, as described in the May 1998 report, EDA investments are indeed economically
                       and statistically significant in determining county employment levels.
                                First, and most importantly, GAO’s analysis is internally inconsistent. The essenceof the
                       GAO argument is that unmeasured county labor market traits that are associated with EDA
                       investments may be important influences on county employment levels. While the researchteam
                       acknowledges that EDA funds do go to larger counties (in terms of employment and population
                       levels), GAO’s analysis does not consider any other potentially relevant characteristics of
                       recipient counties. In addition to their larger-than-average size, counties receiving EDA funding
                       have lower compensation levels per employee than typical U.S. counties. When controls are
                       introduced for both the number of jobs and the compensation levels of jobs in these counties
                       prior to the period being analyzed, the positive statistical and economic significance of EDA
                       funding is confirmed. There can be no doubt that adding controls for prior levels of employment
                       (which GAO does) also requires consideration of prior compensation levels (which GAO does
                       not do). Failure to consider prior compensation levels leads to a “false negative.” In the GAO
                       analysis, EDA investments appear to have no effect on county employment, when in fact they
                       have a positive effect. GAO criticizes .the EDA analysis for failure to include a variable that
                       would better control for county labor market conditions. Yet, GAO’s own analysis also fails to ’
                       control for a critical variable.
                                Second, a closely related point is that the GAO critique of the May 1998 report is very
                       general and lacks the specific statistical techniques to address the alleged deficiencies of the
See comment 6.         EDA analysis. In particular, GAO argues that ‘Taking county-level effects into account is
                       important . . .” The research team agrees and has employed several alternative methods for
                       controlling for these effects. In addition to controlling for prior employment and compensation,
                       the research team has estimated the effect of EDA funds on the growth (as opposed to the level)
                        of county employment over the period 1990-94 and utilized, a strategy called “random effects
                        estimation,” which controls for a wide variety of unmeasured effedts. All of these analysesresult
                        in favorable estimates of the .effects of EDA grants. As indicated by the statistics in Appendix A,


        16                                                GAO/RCED-99-1lR        EDA’s May 1998 Final Report
          ENCLOSURE II                                                                         ENCLOSURE II

                         these results are both robust and recurring: In all of these statistical analyses, the effect of EDA
                         public works investments on county employment is posirive and sratistically significant at high
                         levels of confidence.
                                  Third, the research team notes that while the GAO “Observations” focus on statistical
                         analyses, as indicated earlier, several other methods were utilized by the research team in
 See comment 7.          concluding that the public works program investments have positive employment effects. These
                         methods, including input-output modeling, first-person interviews, and site visits, all supported
                         the statistical regression analysis. In a situation like this, when the preponderance of evidence
                         indicates that EDA funding is positively associatedwith job growth in recipient counties, only a
                         more comprehensive analysis than the one undertaken by GAO could effectively undermine the
                         multiple contributions to these research findings. At a minimum, GAO must perform a complete
                         set of statistical analyses, which clearly has not been done at this point. The next section explains
                         the EDA statistical methods in more detail.

                         Actual Regression Results
                                The following is a restructured modelthat includes both employment and compensation.

                         (1)    EMP,=a+P,       EDA,j+p, EMP++p, PAY,+p

                         (2)    PAY,=y+G,      EII~P,~+~, EMP,+E

                         where EDA is the EDA public works grant amount, EMPr is employment in each county in year
                         t, PAY, is average compensation per employee in year t,i is some positive integer and a proxies
                         for all other determinants of county employment, which will be taken as a constant. In this
See comment 8            specification, p1 is meant to summarize the effect of EDA public works spending on county
                         employment in year t, conditional on employment in year 2-i and year t county compensation
                         levels. County compensation levels are taken as endogenous and are assumed to be determined
                         by current and past levels of employment, among other factors summarized by y. ‘When this
                         model is estimated with j=l (that is, using 1989 employment as an independent variable), the
                         Estimate of p, is negative and statistically insignificant. When a reduced-form version of this
                         model is estimated, p, is positive and statistically significant. What is the source of this
                                 The answer is that, as noted in the May 1998 report, EDA funds tend to be directed to
                         counties with weak, but large, labor markets, and these counties have lower than average levels
                         of compensation per employee. This fact affects the estimates since EDA investments are
                         negarively correlated with county compensation levels, conditional on 1989 employment and the
                         other independent variables in the system. This does not, of course, mean that EDA investments
See comment 9.           reduce wages in recipient counties. The fact that EDA investments are directed toward areaswith
                         low compensation is directly analogous to the observation, made in the GAO statement of facts,
                         that they are directed to counties with high levels of employment. The approaches to controlling
                         for these similar concerns ought to be consistent: if lagged levels of employment are to be
                         included in the employment equation, then lagged levels of compensation ought to be included in
                         the compensation equation.


            17                                               GAO/RCED-99-11R EDA’s May 1998 Final Report
        ENCLOSURE II                                                                             ENCLOSURE II

                              This is the crucial piece of information that reconciles the GAO and EDA analyses. What
                       is most interesting is that the raw correlation between EDA spending and average county
                       compensation is positive, but the correlation conditional on counq size is negative. This suggests
                       that EDA investments go to large counties with low wages. If GAO is to control for the county
                   1   size half of this joint relationship, it must also control for the other half by including lagged
 See comment 10        compensation levels in the list of variables.
                                Replacing the 1989 employment variable (EMP,,) with the natural logarithm of 1989
                       compensation (PAYJ in equation (2) has the effect of reversing the estimated sign on the EDA
                       coefficient (making it positive) and increasing its t-statistic to over 2.0. That is, the statistically
                       significant, positive effect of EDA investment on county employment is reaffirmed in this model.
See comment 11.        This reaffirms that the impact of EDA investment on county labor markets is both economically
                       and statistically significant. The ambiguity emerging from the GAO statement of facts is the
                       result of the incomplete statistical model used in their research.

                       EDA Grants, New Jobs, and the Cost of EDA Job Creation
                               EDA funds projects onty in areas of substantial economic distress. The research team
                       found that EDA projects are located where levels of unemployment and percent of the population
                       below the poverty level are 40 percent higher than state and national averages and where per-
                       capita income is 40 percent ,Iower than state and national levels. In addition, EDA is prohibited
                       by law from extending financial assistanceto establishments relocating from one area to another
See comment 12.        and has internal review procedures that diminish the possibility of funding jobs relocating from
                       outside the region (interregional shifts). In other words, EDA has procedures that would
                       disqualify an employer relocating from Newark, NJ to a rural area in Montana. Further,
                       interviewees in the May ,1997 report indicated that capital facilities assistance would not be a
                       sufficient drawing card to cause an established business to uproot and relocate to an
                       economically depressedrural or urban area within a region (intraregional shifts).
See comment 13.                The cost of job creation per total job created was estimated at $1,115 per job via the
                       original regression analysis (May 1998 report). In the May 1997 report, if the cost per direct job
                       ($3,058) is divided by 1.5 (the multiplier established to produce total jobs in the May 1998
                       report), the cost per total job created is about $2,000. GAO’s estimate indicates an EDA cost per
                       job of about $1,600 for the average amount of an EDA grant. Thus, three different estimating
                       procedures produced a job cost ranging from $1,100 to $2,000. This is a tight range of findings
                        on job costs for these types of analyses.

                       The Strength of the Combined EDA and GAO Analyses
                                In its quest for answers, the GAO analyks has caused the research team to test more
                       controlling variables than were included in the original analysis. This procedure when con-eczly
                       done has reaffirmed the original results and continued to show that EDA funding has a positive
                       and statistically significant relationship to job growth in a county. EDA cost per job in the
                       average county, reflecting the original EDA and GAO analyses, falls within a range of $1,100 to
                       S2,OOO.In the revised model, the cost of job creation in now controlled-for, smalZer and higher-
 See comment 14.       wage areas has risen to about $11,000 per job. What does this mean? EDA job creation costs in
                       smaller and higher-wage counties are higher than they are in larger counties having wage rates
                       that are much lower. EDA has a positive and statistically significant effect in the smaller


         18                                                GAOIRCED-99-1lR         EDA’s May 1998 Final Report
          ENCLOSURE II                                                                                                 ENCLOSURE II

                            counties,but it may cost EDA more to create this effect. Appendix A shows differing costsfrom
                            different model specifications.

                                    This researchproject representsone of the first times-if not the first time-that a federal
                            agency has commissionedan independent evaluation of the impact of its programs.The size and
                            nature of the programs make it difftcult to determinewith certainty the actual impact of relatively
                      _..   small individual projects included within them. The statistical methods available are less than
                            totally precise tools, depending as they do on the assumptionsand variables they employ. As
See commertt    15.         such, within the academiccommunity there is someroom to disagree.The relevant question here
                            is not whether there exists a regression equation that can generatea statistically insignificant
                            coeffkient on the level of EDA spending per job; surely an analyst can find such an equation.
                            The real question is whether the preponderanceof the empirical evidencesuggeststhat the EDA
                            Public Works Program is associatedwith real job creation. Given the fact that virtually all. of the
                            analyses conducted in the EDA study indicate that the program does foster job retention and
                            creation in distressedareas,it seemsthat the answerto this questionmust be “yes.”

                            Fisher. F. 1968a “Embodied Technology and the Existence of Labour and OUQW Aggregates,” Review ofkonomic              Studies.
                                         Vol. 35, pp. 391-412.
                                        l%Bb. “Embodied Technology and the Aggregation of Fixed and Movable Capital Goods.” Review of Economic
                            sIydips.VoI.     35. pp. 417428.
                            A          1969.   ‘The Existence of Aggregate Production Functions,* Econometrica. Vol. 37. No. 4. pp. 553-577.
                            Haughwou~        A. 1998. -Aggregate Production Functions, lnterregionai    Equilibrium, and the Measurement of Inframvcture
                                         Productivity.“Joumul   of Urban Economics, Vol. 44. No. 2, pp. 216-227.                                             (’



           19                                                            GAO/WED-99-11R EDA’s May 1998 Final Report
        ENCLOSURE II                                                                                  ENCLOSURE II

                       Appendix A-EDA           Costs Per Job Under Various Treatments of County-level Effects
                                The following table summarizes the results of the various analyses of county labor
                       markets. Column 1 describes the controls for county-level effects included in the analysis;
                       Column 2 provides the estimation method. With regard to the latter, the methods are either:
                       (I) reduced form least squares with dummy variables (LSDV: for the employment equation
                       only); (2) simultaneous equations (ZSLS: for both employment and compensation equations); or
                       (3) generalized least squares (GLS: random effects estimation on the employment equation only).
                       Column 3 reports the cost per job estimate retrieved from the estimation, with asterisks
                       indicating the degree of confidence that the coefficient is distinguishable from zero in a statistical

                             Column l-Effects Included                   Column Z-Estimation                   Column 3-Point
                                                                                Method                            Estimate
                       1. Urban status, house prices,                  Generalized least squares                    S172***
                          demographic traits, recent labor             W-9
                          market trends, and county random             Reduced Form
                       2. Urban status, house prices,           Least squareswith dummy                             %1,1X**
                          demographic traits, recent labor      variables (LSDV) Reduced
                          market trends, and state fixed        Form
                       3. Urban status, house prices,           Least squareswith dummy                             s9 ,953**lv
                          demographic traits, recent labor      variabies (LSDV) Reduced
                          market trends, state fixed effects,   Form
See comment 16.           and lagged county employment
                       4. Urban status. house m-ices,         I Two-stage least sauares                            $11.166**
                          demographid traits, ;ecent.Iabor      (2SLS) Simultaneous
                          market trends, state fixed effects    Equations
                          lagged county employment (in
                          employment equation), and
                          compensation per employee (in
                          compensation equation)
                       ** Estimated effect is statistically distinguishablefrom zero with at least95% confidence.
                       *** Estimated effect is statisticaIly distinguishablefrom zero witb at least99% confidence.
                       Y The cost per job (and its associatedstandarderror) is identical when calculatedfrom a specification in which the
                       dependentvariable is expressedas~XWZJIin employment since 1989.


          20                                                     GAO/R&ED-99-1lR           EDA’s May 1998 Final Report
ENCLOSURE II                                                                               ENCLOSURE II      ’

       Appendix B-Research’ Team Qualifications

      Research Organizations / Team Members

              Researchfor the Economic Development Administration was undertaken by Rutgers
      University, Center for Urban Policy Research (CUPR); New Jersey Institute of Technology,
      National Center for Transportation and Industrial Productivity (NJIT); Columbia University, Na-
      tional Center for Infrastructure Studies (NCIS); Princeton University, Woodrow Wilson School;
      the National Association of Regional Councils’ Economic Development and Planning Division
      (NARC); the University of Cincinnati, School of Planning and Urban Policy; and Economic
      Modeling Specialists, Inc. (EMSI).

      Rutgers University - Cgterfor    Urban Policy Research (CUPR)

      For nearly three decades,the Center for Urban Policy Research has conducted a broad spectrum
      of urban research.In particular, CUPR has concentrated its efforts in analysis of infrastructure,
      public finance, economic impacts and forecasting, land use, environmental policy, and
      geographic information systems.

      The Center for Urban Policy Research has undertaken economic impact and infrastructure
      studies for the National Academy of Science, National Trust for Historic Preservation,
      Environmental Protection Agency, New York Metropolitan Transportation Commission, States
      of South Carolina and New Jersey, Southeast Michigan Council of Governments, and North
      JerseyTransportation Planning Authority.

      New Jersey Institute of Technology (NJIT) -
      National Centerfor Transportation and Industrial Productivity

      The National Center for Transportation and Industrial Productivity represents a substantial
      investment of the NJIT’s resources and research capacity in activities that are intended to address
      problems of relevance to local governments, the state, and the nation. The National Center’s
      researchinvolves federal and state transportation studies for motor vehicles and transit-based

      Current researchprojects include estimation of multi-modal freight flows in the United States;
      smart sensorsfor freight movement; rail intermodal service planning; pipeline infrastructure
      studies to evaluate and develop criteria for the siting of natural gas and hazardous liquid
      transmission pipelines in proximity to the public in urban areas and in sensitive environments;
      economic and land use impacts of transportation projects; design and construction of prototype
      noise barriers; and seismic retrofitting of major bridges.

      Columbia University - National Centerfor Infrastructure Studies
      The National Center for Infiastntcture Studies was established to research technologies,
      techniques, and materials to improve the productivity and durability of infrastructure facilities in


21                                                   GAOAXED-99-11R         EDA’s May 1998 Final Report
                                                                                                ENCLOSURE II

           urban areas. The Center has performed studies of infrastructure demand and supply with funding
           from federal agencies, states,and major cities.

           The Center has established a preventive maintenance management plan for the bridges of New
           York. developed environmentally responsible guidelines for New York City Bridges, and
           performed extensive destructive and non-destructive testing on many of the nation’s suspension
           bridges. It has recently developed an innovative concrete mixture substituting ground waste

           Princeton University - The Woodrow Wilson School of Public and International Affairs

           The Woodrow Wilson School of Public and International Affairs has more than fifty regular
           faculty members,most of whom have joint appointments with the Departments of Economics,
           Politics, or Sociology. The Woodrow Wilson School has research programs in demography,
           development, domestic policy, international studies, and survey research. The principal research:
           units are the Center of Domestic and Comparative Policy Studies, the Center of International    ’
           Studies, the Office of Population Research, and the Survey Center. The Center for Domestic and
           Comparative Policy Studies has undertaken multiple studies of the economic impacts of public
           works projects.

           National Association of Regional Councils (NARC)
           The National Association of Regional Councils promotes and encourages intergovernmental
           cooperation, recognition of the region as an economic entity, and cooperation among the nation’s
           public, private, and civic sectors. Researchthrusts include the capacity and ability of localities to ,
           undertake economic development.

           University of Cincinnati - School of Planning and Urban Policy
           In the last twenty years, the fact&y of the SchooI of Planning and Urban PoIicy has conducted
           researchon community health, computer simulation and GIS, edge cities/metro-towns,
           environmental managementand policy, housing, inner-city development, international urban
           development, and urban design.

           Economic Modeling Specialists. Inc. (EMSI)

           EMS1 is a consulting firm specializing in regional economic modeling and analysis. EMS1 has
           constructed semi-survey economic models in a variety of settings from small rural communities
           to large and interconnected multistate regions. EMS1 has analyzed issues pertaining to energy
           and natural resource policy, transportation policy, fiscal impacts, fum siting, and a wide variety
           of issues pertaining to regional economic development and land-management planning. EMSI’s
           clients have included the Statesof Hawaii, Utah, and Idaho, the U.S. Forest Service, the U.S.
           Department of the Interior, an assortment of county and city governments, and private firms.

22                                                     GAO/RCED-99-11R EDA’s May 1998 Final Report
 ENCLOSURE II                                                                                 ENCLOSURE II

     Robert W. Burchell, Ph.D. (Rutgers University), Dr. Burchell has served as principal or co-
     principal investigator on ‘more than 60 research contracts in a thirty-year career at Rutgers
     University. He has conducted studies for the Federal Transit Administration, U.S. Department of
     Agriculture, Fannie Mae, U.S. Department of Housing and Urban Development, and other feder-
     al, state, and local agencies.For the last five years, his work has been concentrated in the areas of
     economic impacts and costs of infrastructure development.

     Louis J. Pignataro, D.Sc. (New Jersey Institute of Technology). Dr. Pignataro is Executive
     Director of NJIT’s Institute for Transportation and Distinguished Research Professor of
     Transportation Engineering. He has served as primary investigator for more than 35 sponsored
     researchprojects in a variety of areas,including pipeline infrastructure studies in the New York
     metropolitan area.

     F.H. (Bud) Gr@, Ph.D. (Columbia Universityl. Dr. Griffis has more than 37 years of
     experience in design, construction and maintenance of national and international infrastructure
     systemssuch as program management of the JFK International Airport redevelopment program,
     managementof the design and construction bf Ramon Airbase in Israel, and numerous
     infrastructure design and construction projects in Europe and the Far East.

     Andrew F. Haughwour, Ph.D. (Princeton Universiq). Dr. Haughwout is Assistant Professor of
     Public and International Affairs at the.Woodrow Wilson School and Faculty Associate, Offrce of
     Population Research,atPrinceton University. Professor Haughwout has written about the effects
     of city taxes on fiscal stability, the accumulation of assetsand liabilities by state and local
     governments, and the impacts of infrastructure investments on firms and households. His recent
     work appearsin such publications as Regional Science and Urban Economics. Journal of Urban
     Economics, and the National Tax Association ‘s Papers and Proceedings. His work is at the
     leading edge of his field and complements nicely with standard input-output analysis.

     John W.Epling, D.P.A. (National Association of Regional Councils). Dr. Epling brings to the
     project more than 30 years of experience working for local. regional, and state governments in
     four different stateson issues of economic development, infhastructure investment, urban and
     rural revitalization, and other areas.As the Executive Director of the National Association of
     Regional Councils, he has interacted with elected and appointed officials across the country on
     community and regional development and infrastructure needs.

     M. Henry Robison, Ph.D. (Economic Modeling Specialists, Inc.). Dr. Robison has twenty years
     of experience and numerous significant publications in the field of regional economic impact
     modeling and analysis. He is recognized for theoretical work blending regional input-output and
     spatial trade theory, and for development of community-level input-output modeling and
     analysis. He served for ten years as a faculty member and consultant to the University of Idaho,
     producing a wide array of grants and contract research. He is presently the Senior Research
     Economist at the Center for Business Development and Research, University of Idaho, and the
     Principal ResearchScientist for EMSI.


23                                                    GAOLRCED-99-11R EDA’s May 1998 Final Report
ENCLOSURE II                                                              ENCLOSURE II

The following are GAO’s comments on the Department of Commerce’s letter dated
February 12,1999.

GAO’s Comments

1. It was not our purpose to create another statistical model but rather to evaluate the
   analysis and results contained in EDA’s May 1998.fina.l report. As part of our
   evaluation, we tested alternative specifications of the model used in the EDA report
   and found that for some of the specifications we tested, EDA’s grants had no effect on
   county employment. As a result, we believe that the conclusion in the EDA report
   that EDA’s grants create employment is unwarranted.

2. As we noted in our report, the statistical analysis in the May 1998 report does not
   determine whether the jobs attributed to EDA grants are newly created or relocated;
   it attempts to demonstrate that, holding constant other influences, such as whether
   the county is in a metropolitan area, counties that receive EDA grants have more jobs
   than counties that do not receive the grants. We continue to believe that the May
    1998 report overstates the regression results with statements such as “EDA public
   works invesiments do, in fact, create jobs.” While assurances from applicants and
   certifications from businesses benefiting from EDA’s assistance that businesses will
   not relocate jobs may reflect that these businesses do not plan to move jobs from one
   location to another, the assurances do not account for market impacts that such
   moves might create. For example, if an EDA grant leads to increased jobs in a
   particular industry in one county, then the increased output in that industry may lead
   to lower prices, which, in turn, may result in reduced output and employment

3. See comment 1.

4. We continue to believe that to fully evaluate the impact of EDA grants on
   employment, it is important to consider the impact of EDA grants on employment
   using the results from a broader range of specifications. In response to our report,
   the researchers have analyzed additional specifications of their model.

5. We agree that including prior compensation levels is a good idea, and EDA’s
   comments indicated that its researchers conducted additional analyses that included
   both employment and compensation levels that were not presented in EDA’s May
   1998 report. We did not include compensation levels in our analyses because the May
   1998 report showed few effects attributable to compensation levels. After reviewing
   EDA’s comments, we investigated a number of specifications that take into account
   prior pay as well as prior employment, and we found that in some specifications EDA
   grants were significantly related to employment, while in others they were not.
   Whether or not EDA grants were significant depended on which prior year was used
   and how growth variables were modeled. This confirms our view that the results of
   EDA’s study were highly dependent on the model specification used and that the
   study’s definitive conclusions were unwarranted. Furthermore, we continue to
   believe that it is important to consider the impact of EDA grants on employment using
   the results from a broader range of specifications.

6. EDA’s comments indicate that the research team agrees with us that taking county-
   level effects into consideration is important. However, we do not believe that use of

24                                            GAO/ICED-99-11R   EDA’s May 1998 Final Report
ENCLOSURE II                                                                ENCLOSUREII

      “random effects estimation” is appropriate because it does not control for the bias
     Lintroduced by omitting unmeasured effects that are related to measured effects, such
      as omitting county size, which is related to EDA’s expenditure.

7. Although EDA’s contractors used other methods in addition to statistical analyses, in
   the 1998 report they wrote that the results of their statistical analysis suggest that
   EDA grants significantly influence employment. Our review focused on the
   regression analyses in the May 1998 report, in large part because regression analysis
   is a widely accepted methodology for exploring complex cause and effect
   relationships such as this one. We did not intend to create our own alternative model
   nor to critique ah the methods used by the research team. Our assessment involved
   determining whether the results of the analyses reported were sufficient to justify the
   team’s finding of a significant effect. For reasons discussed elsewhere (see comment
   5), we continue to believe that the researchers’ analyses are not sufficient to reach the
   conclusion that EDA grants significantly, influence employment.

8. We believe that the researchers’ use of 1989 data could be problematic because the
   203 projects being analyzed received their final payment by 1990. Thus, many would
   have been complete, or nearly complete, by 1989 and the projects’ impacts on wages,
   if any, could already be represented in the wage data from that year. We focused on
   1985 and 1986 data in our analyses because most of the projects were approved in
   1985 or later, so it is unlikely that the projects had any measurable impact on
   population or employment in 1986 or before.

9. See comment 5.

10. Although the researchers obtained this result in one specification, as discussed in
    comment 5, we analyzed a variety of specifications and found mixed results.

11. See comment 1.

12. See comment 2.

13. We did not intend for readers to interpret the figure of $1,600 per job created as a
    GAO estimate. Our intent was to show that alternative ways of dealing with certain
    technical issues would yield a range of estimates of the cost per job created. The
    estimated cost per job created depends on both the estimated statistical relationship
    between EDA spending and employment and the method used to derive cost
    estimates from that statistical relationship. Our analysis raises questions about both
    and suggests that substantial uncertainly remains about the cost per job created. This
    uncertainty extends beyond the range of $1,100 to $2,000 per job created

14. See comment 13. EDA’s revised model results in an estimate ofthe cost per job that
    is 10 times higher than its previous estimate of $1,100, further demonstrating the
    uncertainty present in its estimates of the cost per job created. In addition, according
    to EDA’s researchers, their revised model represents the cost of job creation in all
    counties, not in smaller and higher wage areas as stated in the comments.

15. We agree with EDA’s characterization that the real question is whether the
    preponderance of empirical evidence suggests that the EDA public works program is
    associated with job creation. However, the scope of our review was limited to the

25                                             GAOLRCED-99-1lR EDA’s May 1998 Find Report
ENCLOSURE II                                                               ENCLOSURE II

     regression analyses in EDA’s May 1998 report. In our judgment, the results contained
     in that report do not provide sufficient evidence to conclude that EDA grants create
     jobs nor to reliably estimate the cost per job created.

16. According to correspondence with EDA’s contractors subsequent to EDA’s letter,
    “recent labor market trends” were not included in analyses 3 and 4 as indicated in
    column 1 of the table.


26                                              GAOIRCED-99-1lR   EDA’s May 1998 Find Report
Ordering     Information

The first copy of each GAO report and testimony is free.
Additional   copies are $2 each. Orders should be sent to the
following  address, accompanied by a check or money order
made out to the Superintendent     of Documents, when
necessary. VISA and Mastercard      credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address
are discounted    25 percent.

Orders by mail:

U.S. General Accounting     Office
P.O. Box 37050
Washington,  DC 20013

or visit:

Room 1100
700 4th St. NW (corner     of 4th and G Sts. NW)
U.S. General Accounting      Office
Washington,  DC

Orders may also be placed by tailing (202) 512-6000
or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and
testimony.   To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512-6000 using a
touchtone   phone. A recorded menu will provide information       on
how to obtain these lists.

For information on how to access GAO reports on the INTERNET,
send an e-mail message with “info” in the body to:


or visit    GAO’s World Wide Web Home Page at:

United States                            Bulk l&ate
General Accounting  Office         Postage & Fees Paid
Washington, D.C. 20548-0001                 GAO
                                     Permit No. GlOO
Official Business
Penalty for Private    Use $300

Address   Correction   Reauested