United States GAO General Accounting Washington, Office D.C. 20548 Resources, Community, and Economic Development Division B-281059 March 23,1999 The Honorable John R. Kasich Chairman, Committee on the Budget House of Representatives Subject: Economic Develonment: Observations Regarding the Economic Development Administration’s Mav 1998 Final Renort on Its Public Works Program Dear Mr. Chainnan: You requested that we provide you with our observations on the analyses presented in the Economic Development Administration’s (EDA) May 1998 report entitled Public Works Program-Multinlier and Emnlovment-Generating Effects. Final Renort. The report concludes that EDA’s public works program “does indeed produce permanent private- sector employment at a relatively low cost.” We agreed to examine the report’s regression analyses’ to determine whether the report’s results would be similar if additional variables that directly took into account the prior level of a county’s employment or population were included in the analyses. We also agreed to determine whether this report’s job creation estimates include only new jobs or might also include jobs relocated from another area and to examine other technical issues that might affect the reliability of the cost estimates presented. As we stated in our 1996 report, we believe that attempting to quantify the gains from economic development programs is difficult.2 A persuasive study of the impact of a development program would have to demonstrate an improvement in the economy of an area receiving assistance, link that improvement to an agency’s programs, and rule out alternative causes. In our 1996 report, we found no studies that met these criteria. In part because of our report, EDA commissioned a series of studies-including the May 1998 report-on its public works program. ‘Regression analysis is a technique used to estimate the kfluence of individual factors while holding constant, or controlling for, the effects of other factors. The analyses in the EDA report estimated the innuence of EDA grants on employment while controlling for the effects of other factors that influence employment, such as whether a county receiving a grant is in a metropolitan area. 2Economic Development: Limited Information Ekists on the ImDact of Assistance Provided bv Three Agencies (GAO/RCED-96-103,Apr. 3,1996). GAO/RCED-99-11R EDA’s May 1998 Final Report B-281059 RESULTS IN BRIEF We do not believe that the conclusion in the EDA report about the linkage between its public works program and increases in permanent private sector jobs is warranted. Our analyses produced different results from the EDA study. When we included each county’s level of employment prior to the EDA grant in our regression analyses, we found that prior employment levels had a significant effect on counties’ post-grant employment levels, and EDA grants no longer had a significant effect. By not taking into account the counties’ previous employment levels, the study may have shown only that larger counties-with high levels of employment-tend to receive the most grants. In addition, the job creation estimates did not address whether the jobs are new or relocated from another area but only that counties receiving EDA grants have more jobs than counties not receiving grants. We also identified some technical issues pertaining to the certainty of the cost estimate per job created. For example, the report’s conclusion that an additional $10,000 in EDA spending in a typical county would yield nine new jobs is applicable only if that county were receiving a grant substantially smaller than the average grant EDA made. BACKGROUND EDA was established in 1965 within the Department of Commerce to target federal resources to economically distressed rural and urban communities across the country that lagged in economic development, industrial growth, and personal income. EDA’s budget authority for fiscal year 1998 was $340 million, and its obligations for its public works grants were $178 million. The public works and development facilities program, EDA’s program with the largest amount of funding, provides grants to help distressed communities attract new industry, encourage business expansion, diversify economies, and generate long-term private sector jobs. Selection criteria for public works grantees favor distressed areas with high unemployment; however, these areas may be within counties that do not have overall levels of distress or high unemployment. EDA commissioned two studies on the effect of its public works funding on job creation. The first study, entitled Public Works Program: Performance Evaluation, Final Renort (May 1997), used a combination of surveys, seminars for grantees, and site visits to obtain performance information from 203 EDA projects that received their final payment in 1990. The study found that 96 percent of the projects produced permanent jobs 6 years after completion at a cost to EDA of $3,058 per job created or retained. The second study, entitled Public Works Program-Multinlier and Emnlovment-Generating Effects. Final Renort (May 1998), employed two techniques to document the effects of EDA’s public works projects on the employment growth of counties receiving grants. The study builds on the findings from the first study. Using the first technique, researchers found that for every 10 jobs that can be directly attributed to an EDA project, an additional 5 jobs will be created or moved to the county in which the project is placed. The second technique, regression analysis, showed that $10,000 increments in public works grant spending produce, on average, nine jobs, for an estimated cost of $1,100 per job created or moved to a county that receives an EDA grant. 2 GAO/RCED-99-IlR EDA’s May 1998 Final Report B-281059 ALTERNATIVE ANALYSES INCLUDING ADDITIONAL VARIABLES PRODUCE DIFFERENT RESULTS We believe that the results of EDA’s study are highly dependent on the model specification used and that the study’s definitive conclusions are unwarranted. By changing the specification3 of the model to control for the effect of prior county employment, we obtained substantially different results that show that EDA expenditures did not have a significant effect. To fully evaluate the impact of EDA grants on employment, it is important to consider the impact using the results from a broader range of specifications. EDA’s study used county wages, house prices, urban status, the recent performance of the labor market (for example, the growth of employment from 1988 through 1990), and the demographic characteristics of the population as factors to take into account in estimating the effects of EDA grants.’ Taking these factors into account, the researchers found that EDA’s expenditures have a statistically significant, positive effect on county employment levels. However, we do not believe that the study sufficiently accounted for county-level effects, such as the prior levels of employment and the population, that could also be associated with current employment levels. Taking county-level effects into account is important because EDA’s spending in 1990 was associated with a county’s prior levels of employment and population- Without accounting for these effects, EDA’s study may be showing only that larger counties with higher levels of employment tend to receive the most grants-rather than measuring the impact of EDA’s grants on employment growth. When we examined the placement of the 203 EDA projects in this study, we found that the average county receiving a grant had a 1986 employment level of about 58,000 and a 1985 population of about 162,000. However, the average county not receiving a grant had an average 1986 employment level of about 24,600, and a 1985 population of about 70,500. Examinin g the most populous county in each state, we found that 24 percent received one (or more) public works grants; 6 percent that were neither the largest nor the smallest in population received one or more grants; and only 2 percent of the counties with the smallest population in each state received a grant. We conducted several analyses using additional variables-such as county employment and population-that were not included in EDA’s study. These analyses enabled us to assess whether EDA grants would still have a significant effect in regression analyses that took into account the effects of these additional factors on employment. For example, when we used employment levels in 1986 as a factor in our regression analyses, the analyses showed that EDA’s expenditures in 1990 had no significant effect on employment levels for 1990 through 1994 in the counties receiving grants. Including the county’s population in 1985 instead of county employment as a factor, we also found that EDA’s expenditures %y model specification we mean the specific set of variables included in the model and the specific assumptions made about how those variables are related to each other. ‘In addition to these factors, the researchers took into account factors particular to the state economy, which are sometimes called state fixed-effect variables. 3 GAOLRCED-99-11R EDA’s May 1998 Final Report B-281059 had no effect on employment levels. We chose to use 1985 population and March 1986 employment data because most of the projects in the analyses were approved in 1985 or later. Therefore, it is unlikely that the projects had any measurable impact on population or employment in March 1986 or before.5 At the suggestion of the EDA contractors, we also controlled for 1988 employment. Again, we found that EDA expenditures had no significant effect on employment levels. (See enc. I for the detailed specifications and results of our analyses.) JOB CREATION ESTlMATES DO NOT ADDRESS WHETHER JOBS ARE NEW OR RELOCATED EDA’s regression analyses did not address whether the jobs attributed to EDA grants are new or relocated. The analyses attempted to demonstrate that, holding constant other influences, such as whether the county is in a metropolitan area, counties that receive EDA grants had more jobs than counties that did not. As a result, the job impact in a particular county that is measured in the study may include newly created jobs in that county, jobs that moved to the county from another location, or jobs that would have moved to another county but were retained because of the grant. Because the analyses did not measure whether these jobs are new or simply relocated, some statements in the report overstate the regression results. For instance, on page 43, the report states, “EDA public works investments do, in fact, create jobs,” and on page 45, it states, “the EDA Public Works Program is a significant (in both a statistical and numerical sense) contributor to the productive capacity of the country.” While the regulations for the public works program prohibit the use of grants to relocate jobs from one area to another, EDA relies on assurances from applicants and certifications from businesses benefiting from the assistance that the businesses will not make such transfers. However, these assurances do not account for market impacts. Even if a county receiving an EDA grant has additional jobs because of that grant, the economy as a whole may not have necessarily gained jobs. For example, if an EDA grant leads to increased jobs in a particular industry in one county, then the increased output in that industry may lead to lower prices, which, in turn, may result in reduced output and employment elsewhere. TECHNICAL ISSUES ABOUT THE RELATIONSHIP BETWEEN EDA SPENDING AND EMPLOYMENT MAY AFFECT COST ESTIMATES We identified two additional technical issues that could affect the estimates of the cost per job presented in EDA’s 1998 report. First, to use their model specification, EDA’s contractors needed to make an arbitrary decision on how to include the key predictor variable-the level of EDA’s expenditures-for the large majority of counties that did not receive grants. We found that by using different arbitrary adjustments of that variable but otherwise following the method used in EDA’s report, we obtained different estimates- both higher and lower-of the cost per job created.” For example, over the range of ‘According to EDA’s contractors, about 5 percent of the projects were completed before 1986. “In their andyses, EDA’s contractors used a specilkation in which they used the natural logarithm of the 4 GAO/RCED-99-11R EDA’s May 1998 Find Report ’ B-281059 ahernatives we used, we obtained estimates of the cost per job created ranging from about $600 to about $1,600, compared with the estimate of $1,100 in EDA’s report. - Second, the report’s conclusion that an additional $10,000 in EDA spending in a typical county would yield nine new jobs is applicable only if that county were receiving a grant substantialIy smaIIer than the average grant EDA made. The model specification that EDA’s contractors used implies that the number of new jobs resulting from a specified increase in EDA spending (such as $10,000) is different at different levels of spending.7 EDA’s contractors estimated the effect of a $10,000 spending increase when the grant level was about $46,500.’ However, the average grant during this period was about $750,000. We found that the estimates in EDA’s report implied that a $10,000 increase in EDA spending in a county receiving an average grant, and with average employment among counties receiving grants, would yield about six new jobs, rather than the nine new jobs that the EDA report concluded would be created. This finding of six new jobs would raise the estimated cost per job created from about $1,100 to about $1,600.’ AGENCY COMMENTS AND OUR EVALUATION We provided a draft of this report to the Department of Commerce for review and comment. The Department’s Economic Development Administration (EDA) provided comments on the report. EDA disagreed with our findings regarding the effect of its variables, rather than the variables themselves, in the regression analyses. The natural logarithm of a number is the power to which e (approximately 2.7) must be raised to produce that number. When they created the logarithm for EDA’s expenditures, they added a value of 1 x 10 p only to the counties that did not receive a grant (and therefore had a value of zero), in effect treating those counties as if they received grants equal to a . small fraction of one cent. Some adjustment was necessary because the logarithm of zero is undefined However, the choice of 1 x 10” was arbitrary, and when we used adjustments ranging from 1 x 10 .’ to 1 x lo’@ instead, we obtained estimates of the cost per job ranging from a little more than 50 percent of the estimate in EDA’s report to nearly 50 percent more than the estimate in the report. An alternative approach to dealing with this situation is to add 1 to the level of EDA expenditure for each observation before creating the logarithm. ?.f the contractors had used a linear specification, then the estimated change in employment for a $10,000 grant increase would be the same regardless of the initial grant level. However, in a logarithmic specification such as the one used in EDA’s report, the estimated percentage change in employment would be the same for any given percentage change in EDA spending but the absolute change for a given absolute change in spending would vary. 8The contractors chose this level because it represented the mean grant value in the data set used for estimation, which includes counties not receiving grants (total EDA grant spending divided by the number of counties). Because most counties did not receive grants during this period, this number is substantially lower than the average grant EDA made. ‘When the contractors estimated the number of new jobs created from a $10,000increase in EDA spending, they did not use a measure of employment that appropriately corresponded to the $46,500measure they used for grant size. If we had used EDA’s method of selecting an employment level to correspond with a grant level, we would have found that the estimates in EDA’s report imply that a $10,000increase to a $750,000grant would create about 1.5 new jobs at a cost per job created of about $6,700. 5 GAOLRCED-99-1lR EDA’s May 1998 Final Report B-281059 investments on county employment. EDA stated that our findings were based on an incomplete statistical model. It said that if our economic model took into account prior compensation levels for jobs in each county as well as prior employment, its grants would be shown to have a positive effect on employment. EDA’s comments mischaracterized our findings. It was not our objective to create another model but r&her to evaluate the analyses and results contained in EDA’s May 1998 final report on the public works program. We did not take into account prior compensation levels for jobs in each county in our analyses because EDA’s May 1998 report showed few effects attributable to compensation levels. After reviewing EDA’s comments, we investigated a number of model specifications that took into account prior compensation levels as well as prior employment. We found that under certain assumptions EDA grants had significant effects on employment while in others they did not. Whether or not EDA grants had a significant effect depended on which prior year’s data were used and how growth in wages and employment were modeled. This inconsistency confirms our view that the results of EDA’s study were highly dependent on the model specification used and that the study’s definitive conclusions were unwarranted. Therefore, we made no changes to the report. EDA also disagreed with the report’s discussion on job creation. EDA said that its grants create new jobs in highly distressed local economies and that, because of legislative and regulatory restrictions, new jobs resulting from job relocations are unlikely. EDA’s comments on the relocation issue cover only part of our concern about the job creation estimates in EDA’s report. Our main concern is that EDA’s analyses do not address whether the jobs attributed to its grants come at the expense of jobs elsewhere in the economy. Therefore, some statements in the EDA report indicating that EDA investments do create jobs overstate the results of the regression analyses. We made no changes to the report in response to EDA’s comments. EDA’s comments and our detailed responses appear in enclosure II. OBJECTIVES. SCOPE. AND METHODOLOGY In order to examine the regression analyses in EDA’s report, we obtained the data set used by EDA’s contractors. We then recreated some of the regression results of EDA’s contractors to confirm our understanding of the data set. Next, we added employment data from the 1986 and 1988 Countv Business Patterns and 1985 population data from the Bureau of the Census. We used these additional data to test alternative specifications to examine whether similar findings regarding the association between the level of EDA grants in a county and that county’s employment level would be obtained using these alternatives. A detailed description of our methodology appears in enclosure I. To examine other technical issues that might affect the reliability of the cost estimates, we reviewed the report’s methodology. As we identified areas of concern, we assessed their implications. We also assessed the methodology to determine if it could be used to identify whether jobs associated with EDA grants were necessarily jobs that had been newly created. 6 GAO/RCED-99-1lR EDA’s May 1998 Find Report B-281059 We are sending copies of this report to Senator Robert C. Byrd, Senator Joseph I. Lieberman Senator Ted Stevens, and Senator Fred Thompson and to Representative Dan Burton, Representative David R. Obey, Representative Henry A. Waxman, and Representative C.W. Bill Young in their capacities as Chair or Ranking Minority Member of Senate and House Committees. We are also sending copies of this report to The Honorable William M. Daley, the Secretary of Commerce. Copies will also be made available to others on request. If you or your staff have any questions about this report, please call me on (202) 512-7631. Major contributors to this report were Susan Campbell, Jay Cherlow, and Austin Kelly. Sincerely yours, Judy A. England-Joseph Director, Housing and Community Development Issues Enclosures - 2 GAO/RCED-99-11R EDA’s May 1998 Final Report ENCLOSURE I ENCLOSURE I RESULTS OF OUR ANALYSIS This enclosure describes the alternative model specifications we tested, the variables we used, and the results of several regressions we ran. Our purpose was to examine whether the Economic Development Administration’s (EDA) conclusion; that a statistically significant association exists, between the level of EDA grants in a county and that county’s employment level, would be reached using alternative model spectications. In particular, we tested whether this statistically significant association remained once we controlled for the county’s level of employment at an earlier date or for the county’s population. We found that in alI of the specifications we used, the relationship between EDA grants a county received and the employment level in that county was not statistically different from zero. That is, these results suggest that (1) the report’s findings are highly sensitive to the specification of the model used and (2) the signiscant relationship found when the prior county employment level is not taken into account may reflect the fact that the highest levels of EDA grants go to the counties with the highest employment levels rather than that there is any causal relationship between EDA grants and employment. l MODEL SPECIFICATIONS EDA’s contractors used a database that included the total grant amount for EDA projects receiving final funding in 1990 for each county in the United States. They used this database to run several speciiications of a two-equation, pooled, cross-section, time- series model; that is, observations on each county for each of the 5 years were pooled, or combined. The dependent variables were the natural logarithms of the levels of employment and payroll per worker (in 1990 dollars). We tested several specifications that were modifications of the report’s “specification 3.” These specifications, which employ the two-stage least squares (2SLS)’ regression procedure used in specilication 3 of the report, were ail designed to test whether a county’s level of EDA grants continue to show a statistically significant association with the county’s employment level once we control for the county’s prior level of employment or population.’ We believe that by omitting this factor, the report’s results may overstate the importance of EDA grants on a county’s employment level because of the association between EDA grant spending in 1990 and prior levels of employment and population. ‘2SLSis an econometric technique used when a model contains two or more dependent variables, in this case, employment and pay, that simultaneously infhrence each other and are thereby jointly determined. 2Weused specification 3 as the basis for our alternatives for two reasons: (1) the results from specification 3 are the ones EDA uses to estimate the cost per job and (2) specification 3 is the only alternative presented that considers a county’s compensation level as a factor explaining employment level, although the report says that “clearly, a well-specified model of employment must include compensation.” In subsequent correspondence, EDA’s researchers have suggested that despite what the report says, regression using reduced form ordinary least squares @IS) rather than 2SLSmay be more appropriate. Unlike 2SIS, OIS regression does not treat the dependent variables, compensation and employment, as being jointly determined. Since we were asked to review the analyses used in the report as the basis for the conclusion about the impact of EDA grants, and the report makes a strong casefor using 2SI.Sbecause of the complex relationship between compensation and employment levels, we used 2SLSin our specifications. Moreover, even if regressions using OLS that control for prior employment or population levels found that the level of EDA grants remained a statisticaIly significant factor, the results from 2SLSregressions would remain important in assessingthe impact of EDA grants. 8 GAOIRCED-99-11R EDA’s May 1998 Final Report ENCLOSURE I ENCLOSURE I We used several specifications to estimate the significance of EDA grants while controlling for prior employment or population levels.3 Because we wanted to measure employment before the grants could have had an impact, we used the employment levels in 1986 as a control variable. We conducted two regression analyses including the natural logarithm of that variable, one that included all the variables in the report’s specification 3 and one that omitted the variable measuring growth in employment from 1988 to 1990. Although using a‘later year would seem to run a greater risk that some of the grant projects might already have been completed and begun having an impact, at the suggestion of EDA’s contractors, we also used a specification that used the employment level in 1988. In that specification, we omitted the growth in employment from 1988 to 1990 because the two variables together would equal 1990 employment. Finally, we also used a specification in which we used the natural logarithm of 1985 population as a control variable. Population is a good, although not perfect proxy for employment, and is almost certainly not substantially affected by the grants themselves. 3Weestimated a regression that was identical to specification 3 to confirm that we understood the data set well enough to replicate EDA’s contractors’ results. Our coefficient estimates were virtually identical to those in the EDA report. GAOLRCED-99-11R EDA’s May 1998 Final Report ENCLOSURE I ENCLOSURE I VARIABLE DESCRIPTIONS Table I.1 summarizes the dependent and independent-or predictor-variables that were used in the analysis along with their corresponding means. Table 1.1: Definitions and Mean Values of Variables Used in the Analvsis Variable 1 Definition 1 Mean Detsendent variable LTEMPMM” The natural logarithm of 8.68 county employment in the relevant year,. 1990-l 994 Predictor variable SMSHARE 1The proportion of the 79.32 county’& firms with fewer than 10 employees BIGSHARE The proportion of the 0.04 county’s firms that have more than 1,000 employees URBAN Dummy variable equals 1 if 0.27 a county is in a Metropolitan Statistical Area (MSA); otherwise it is zero LHVSO The natural logarithm of 10.79 - 1990 county house value GREM8890 The percentage growth in 0.03 ( county employment from 1 I 1988 to 1990 GRWA8890 The percentage growth in -0.02 payroll per county worker from 1988 to 1990 LPAY The natural logarithm of 9.70 county payroll per worker LEM P86. The natural logarithm of 8.53 1986 county employment LEMP88 The natural logarithm of 8.64 1988 county employment LPOP85 The natural logarithm of 10.14 1985 county population LEDA The natural logarithm of the -18.69 county’s EDA grant value in , ;:9Tlevant year, 1990- , I “The natural logarithm of employment was used as a predictor variable in the model’s other equation that had county payroll per worker as the dependent variable. 10 GAOIRCED-99-11R EDA’s May 1998 Final Report ENCLOSURE I ENCLOSURE I REGRESSION RESULTS Table I.2 shows the second-stage regression results for EDA’s specification 3 with the logarithm of employment as the dependent variable and the four alternative specifications we tested.4 In each case, the coeffkient on our measure of prior employment or population level is highly significant, while the coefficient on our measure of the value of EDA’s grants is not statisticaZly different-or even close-from zero at conventional levels. Table 1.2: Reqression Results for EDA’s Specification 3 and Our Four Alternative SDecifications . 1 Coefficient estimates and standard errors’ Predictor EDA’s GAO’s GAO’s GAO’s GAO’s variable Spec. 3 Alt. 1 Alt. 2 Ah. 3 Alt. 4 1 SMSHARE I -0.112b 1 -0.025b 1 -0.025b 1 -O.OOgb1 -0.093b 1 (0.008) (0.003) (0.003) (0.001) (0.011) BIGSHARE 0.941 b 0.825b 0.908b 0.347b 2.983b (0.235) (0.148) (0.184) (0.063) (0.521) URBAN o.454b 0.068b 0.070b 0.021 b -0.141b (0.028) (0.011) (0.012) (0.004) (0.033) LHVSO 0.962b 0.428b 0.44gb 0.1 32b 1 .252b (0.107) (0.0522) (0.062) (0.021) (0.190) GREM8890 0.266b -0.064b --- --- w-- (0.015) (0.020) . GRWA8890 -0.835b 0.951 b 0.842b 0.248b 2.590b (0.305) (0.248) (0.229) (0.077) (0.605) LPAY 1 .422b ‘-2.63b -2.762b -0.764b -7.434b (0.775) (0.621) (0.703) (0.232) (1.797) LEDA 0.0074b 0.0001 -0.00005 -0.00009 -0.0016 (0.0007) (0.0005) (0.0005) (0.0002) (0.0013) LEMP86 --i 1 .054b 1 .0003b --- --- (0.034) (0.019) LEMP88 m-e --- --- 0.99tsb --- (0.006) LPOP85 --- -we m-m --- 1 .207b (0.038) Summary statistics Observations 15,591 15,552 15,552 15,592 15,587 Adjusted 0.8545 0.944 0.937 .0.991 0.665 R-square ‘All regressions include year and state fixed effects, the results of which are available upon request. %tatistically different from zero at the 99 percent confidence level. 4~though our emphasis was on the effect of EDA gi-ants on county employment, because we were using modifications of the contractor’s 2SIS estimation procedure, we also estimated specification 3’s payrou equation with similar modifications. We have not reported those results here, but they are available on request. 11 GAO/RCED-99-1lR EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II COMMENTS FROM THE DEPARTMENT OF COMMERCE Note: GAO comments supplementing those in the report appear at the end of this enclosure THE SECRETARY OF COMMERCE Washington. OX. 20230 FEE I 2 I%cl Ms. Judy A. England-Joseph Director Housing and Community Development Issues U.S. General Accounting Office Washington, DC 20548 Dear Ms. England-Joseph: Thank you for your letter requesting the Department’s comments on the draft General Accounting Office (GAO) report entitled “Economic Development: Observations Regarding the Economic Development Administration’s (EDA) May 1998 Final Report on its Public Works Program.” We appreciate the opportunity to provide comments, which support the results of the independent evaluation of the EDA Public Works program. EDA management and professional staff and members of the research team have reviewed the report, and worked with GAO to review the methodologies used in the evaluation. The EDA- commissioned research project represents management’s strong commitment to program evaluation. Comprehensive comments from the Assistant Secretary for Economic Development and the Rutgers University research team are attached. We appreciate your interest in the Department and its programs. Attachment 12 GAOLRCED-99-1lR EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II UNITED STATES DEPARTMENTOF COMMERCE Thu Assistant Sam-etary for Economic Dsvalopment Washragton. DC. 20230 MEMORANDUM FOR THE SECRETARY --I f- . .A- FRO-M: Phillip A. Singennan,Ph.D( -&a ‘( j&,.;c.Aw-- Assistant Secretary ; - : for EconomicDevelopment SUBJECT: GAO Review of EDA’s May 1998Final Report on its Public Works Program We appreciate the opportunity to respondto the GAO review of EDA’s Public Works program evaluation. Over the past six months EDA, GAO, and EDA’s researchteams have worked togetherto review the methodoIogiesusedin the evaluations. In addition, we take this oppommity to thank the GAO staff for their collaboration with us on our program evaluation efforts and on our commitment to fully comply with the requirementsof the Government Performanceand Results Act of 1993. Two researchteams,both led by Dr. Robert W. Burchell of Rutgers University, conducted studiesof EDA’s Public.Works program resulting in the 1997and 1998reports. In this work, they undertookground-breaking researchin evaluating the impact of EDA’s major program. Their evaluations,using several independentmethodologies,have confirmed the effectiveness of EDA’s investmentsin creating permanentjobs. The researchconfirmed that EDA projecis createsubstantialnumbersof jobs and do so in areasof significant distress,areaswhere the averageunemploymentlevels, percent of the population in poverty, and per capita income were 40 percent worse than stateand national averages..Furthermore,every Sl million of EDA riding leveraged 510.08 million in private sectorinvestmentand increasedthe local tax base byS10.13 million. However, the estimatedaverage cost of thesejobs varies slightly between the two Rutgers t,:ams’ methodologiesand between Rutgersand GAO. Basedon a detailed analysis of 100 percent of the Public Works projects completedin FY 1990-more than 200 projects-the first researchteam conlirmed that EDA projectscreateddirectjobs at an average cost of about S3,OOO.The secondresearchteam found slightly lower job-cost estimatesusing econometric techniques. EDA’s job creation costsvary dependingon the size and income levels of an area: in larger and poorer counties the costsaremuch lower than in smaller countieswhere wage rates are higher. Both the se&d Rutgers team and GAO employ standard,academically acceptedstatistical methodologies. However, the methodsavailable to the.researcherare less than totally precise tools, depending as they do on the assumptionsand variables they employ. As such, within the academiccommunity there is some room to disagree. After careful consideration and following consultation with the Rutgers researchteam that conductedthe econometricstudies,we must 13 GAOIRCED-99-1lR EPA’s May 1998 Final Report .. -.. ENCLOSURE II ENCLOSURE II respectfully and firmly disagreewith GAO’s observations regarding the effect of EDA investments on county employment. We believe conclusions drawn by GAO are based on an See Comment 1. incomplete statistical model used in their research. In fact, if properly developed, the data support the conciusion that the effect of EDA Public Works investments on county employment is p&rive and sraristicaI[v significunf at high levels of confidence. The research team’s review of the GAO methodology is set forth in the Attachment. EDA funding createsjobs in highly distressed local economies. These indeed are new jobs, not ones relocated from elsewhere. The high level of economic distress in assisted areas, EDA’s See comment 2. legislative exhortation against funding job relocations, as well as EDA’s internal implementing procedures make funding job relocations most unlikely. Furthermore, research undertaken by economists such as David Birch and others, has shown that the number of jobs in any region created by the relocation of industries or firms is quite insignificant. The Rutgers researchteam’s review of the underlying methodology is attached. EDA fully endorses its methodologies and findings. Attachment 14 GAO/RCED-99-UR EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II Attachment RUTGERS RESEARCH TEAM RESPONSE TO GAO’S “OBSERVATIONS REGARDING THE ECONOMIC DEVELOPMENT ADMINISTRATION’S MAY 1998 FINAL REPORT ON ITS PUBLIC WORKS PROGRAM” Introduction The research team appreciates GAO’S diligence in pursuing this important question, its willingness to provide us with the details of its analyses, and the opportunity to respond in this See coqment 3. document. The research ‘team respectfully disagrees, however, with the conclusions GAO draws regarding the effect of EDA investments on county employment. The conclusions drawn by GAO are based on an incomplete statistical model, as will be explained below. First, some background information on the various EDA studies and their results. Background on the EDA Public Works Reports Two studies and reports were completed on the impacts of EDA funding on job creation: The first report, Public Works Program: Performance Evaluation (May 1997), found that 96 percent of 203 EDA projects produced permanent employment in the location of their development at a cost of $3,058 per direct job. This information was received from the field through both telephone interviews and responses to a written information request. One hundred percent of the projects were contacted and provided information. Thirty percent of the sites (60 projects) were actually visited by research team members, who verified fast-hand the information reported by the survey. Upon visiting the sites, research team members entered buildings and counted jobs. Further, they interviewed the owners of the facilities and the EDA grantees at the site. Btith business owners and EDA grantees agreed that these were not locations where jobs would grow spontaneously-levels of unemployment and families living below the poverty line were 40percent worse than state and national averages. EDA$nding was reported as “critical” in creating new jobs in these areas. The second report, Public Works Program: Multiplier and Emplo-vment-Generating Eficts (May 1998), sought to determine whether EDA projects had an effect on job growth in counties where this funding was present. The 203 public w.orks projects in 172 counties were viewed in the context of jobs being created in 3,200 counties nationally. Two econometric techniques were used: hput-output analysis found that EDA-funded direct permanent empioy- ment produced indirect and induced permanent employment at a ratio of 2 to 1; that is, the employment multiplier is 1.50. (Direct private-sector project investments similarly leverage indirect and induced private investment under a comparable investment multiplier of 1.44.) The regression analysis found that EDA funding had a statistically significant and positive effect on county total employment levels. Further, counting aNjob growth (total jobs), EDA funding led to jobs created at a cost of about $ 1, 115 per total job. This is roughly comparable to the finding that EDA created direct jobs at a cost of $3,058 per direct job, since the total nlunber of jobs related to EDA funding is likely to be 1.5 times the number of direct jobs. Dividing $3,058 by 1.5 indicates a total job cost of about $2,000 per job. One source of data (regression analysis) revealed that EDA created jobs at a cost of approximately $I,1 15 per job; a separate independent source of data (survey and physical counting) confirmed that EDA created jobs at a cost of about $2,000 per total job. The procedures employed in the May I998 regression analysis are standard econometric tests widely 15 GAO/NED-99-11R EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II employed in academic and practitioner circles for complex cause and effect relationships of the kind evaluated here. They are sophisticated and complete. yielding results with which the See comment 4. research team is comfortable. Further, their results basically paraUe1 findings from the Ma) I997 stud\!, wherein these new jobs were actually counted. The fact that tx’o vety dtfferent methodologies generated very similar job costfigures is important and noteworthy. The GAO Statement of Facts, “Observations on EDA’s May 1998 Report” i GAO claims that the relationship between EDA public works funding and county employment levels is statistically insignificant. This point is crucial, as it goes to the heart of the matter: do EDA investments have a discernible effect on county employment? The researchteam is convinced that the GAO conclusion is incorrect and that there is a positive relationship between EDA investments and county employment, as stated in the May 1998 report. The GAO analysis begins with the observation that EDA public works grants are not randomly distributed across counties, and that failure to control for the unique characteristics of these counties leads to a false positive relationship between EDA funds and county employment. See comment 5. GAO’s analysis emphasizes the fact that EDA funds are provided to counties that are, on average, larger (in terms of their employment and population levels) than typical U.S. counties. GAO argues that once this fact is acknowledged and controlled for with statistical techniques, there is no relationship between EDA funds and county employment levels. The GAO conclusion does not stand up to scrutiny, however. Three considerations lead the research team to the : conclusion that, as described in the May 1998 report, EDA investments are indeed economically and statistically significant in determining county employment levels. First, and most importantly, GAO’s analysis is internally inconsistent. The essenceof the GAO argument is that unmeasured county labor market traits that are associated with EDA investments may be important influences on county employment levels. While the researchteam acknowledges that EDA funds do go to larger counties (in terms of employment and population levels), GAO’s analysis does not consider any other potentially relevant characteristics of recipient counties. In addition to their larger-than-average size, counties receiving EDA funding have lower compensation levels per employee than typical U.S. counties. When controls are introduced for both the number of jobs and the compensation levels of jobs in these counties prior to the period being analyzed, the positive statistical and economic significance of EDA funding is confirmed. There can be no doubt that adding controls for prior levels of employment (which GAO does) also requires consideration of prior compensation levels (which GAO does not do). Failure to consider prior compensation levels leads to a “false negative.” In the GAO analysis, EDA investments appear to have no effect on county employment, when in fact they have a positive effect. GAO criticizes .the EDA analysis for failure to include a variable that would better control for county labor market conditions. Yet, GAO’s own analysis also fails to ’ control for a critical variable. Second, a closely related point is that the GAO critique of the May 1998 report is very general and lacks the specific statistical techniques to address the alleged deficiencies of the See comment 6. EDA analysis. In particular, GAO argues that ‘Taking county-level effects into account is important . . .” The research team agrees and has employed several alternative methods for controlling for these effects. In addition to controlling for prior employment and compensation, the research team has estimated the effect of EDA funds on the growth (as opposed to the level) of county employment over the period 1990-94 and utilized, a strategy called “random effects estimation,” which controls for a wide variety of unmeasured effedts. All of these analysesresult in favorable estimates of the .effects of EDA grants. As indicated by the statistics in Appendix A, 2 16 GAO/RCED-99-1lR EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II these results are both robust and recurring: In all of these statistical analyses, the effect of EDA public works investments on county employment is posirive and sratistically significant at high levels of confidence. Third, the research team notes that while the GAO “Observations” focus on statistical analyses, as indicated earlier, several other methods were utilized by the research team in See comment 7. concluding that the public works program investments have positive employment effects. These methods, including input-output modeling, first-person interviews, and site visits, all supported the statistical regression analysis. In a situation like this, when the preponderance of evidence indicates that EDA funding is positively associatedwith job growth in recipient counties, only a more comprehensive analysis than the one undertaken by GAO could effectively undermine the multiple contributions to these research findings. At a minimum, GAO must perform a complete set of statistical analyses, which clearly has not been done at this point. The next section explains the EDA statistical methods in more detail. Actual Regression Results The following is a restructured modelthat includes both employment and compensation. (1) EMP,=a+P, EDA,j+p, EMP++p, PAY,+p (2) PAY,=y+G, EII~P,~+~, EMP,+E where EDA is the EDA public works grant amount, EMPr is employment in each county in year t, PAY, is average compensation per employee in year t,i is some positive integer and a proxies for all other determinants of county employment, which will be taken as a constant. In this See comment 8 specification, p1 is meant to summarize the effect of EDA public works spending on county employment in year t, conditional on employment in year 2-i and year t county compensation levels. County compensation levels are taken as endogenous and are assumed to be determined by current and past levels of employment, among other factors summarized by y. ‘When this model is estimated with j=l (that is, using 1989 employment as an independent variable), the Estimate of p, is negative and statistically insignificant. When a reduced-form version of this model is estimated, p, is positive and statistically significant. What is the source of this difference? The answer is that, as noted in the May 1998 report, EDA funds tend to be directed to counties with weak, but large, labor markets, and these counties have lower than average levels of compensation per employee. This fact affects the estimates since EDA investments are negarively correlated with county compensation levels, conditional on 1989 employment and the other independent variables in the system. This does not, of course, mean that EDA investments See comment 9. reduce wages in recipient counties. The fact that EDA investments are directed toward areaswith low compensation is directly analogous to the observation, made in the GAO statement of facts, that they are directed to counties with high levels of employment. The approaches to controlling for these similar concerns ought to be consistent: if lagged levels of employment are to be included in the employment equation, then lagged levels of compensation ought to be included in the compensation equation. 3 17 GAO/RCED-99-11R EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II This is the crucial piece of information that reconciles the GAO and EDA analyses. What is most interesting is that the raw correlation between EDA spending and average county compensation is positive, but the correlation conditional on counq size is negative. This suggests that EDA investments go to large counties with low wages. If GAO is to control for the county 1 size half of this joint relationship, it must also control for the other half by including lagged See comment 10 compensation levels in the list of variables. Replacing the 1989 employment variable (EMP,,) with the natural logarithm of 1989 compensation (PAYJ in equation (2) has the effect of reversing the estimated sign on the EDA coefficient (making it positive) and increasing its t-statistic to over 2.0. That is, the statistically significant, positive effect of EDA investment on county employment is reaffirmed in this model. See comment 11. This reaffirms that the impact of EDA investment on county labor markets is both economically and statistically significant. The ambiguity emerging from the GAO statement of facts is the result of the incomplete statistical model used in their research. EDA Grants, New Jobs, and the Cost of EDA Job Creation EDA funds projects onty in areas of substantial economic distress. The research team found that EDA projects are located where levels of unemployment and percent of the population below the poverty level are 40 percent higher than state and national averages and where per- capita income is 40 percent ,Iower than state and national levels. In addition, EDA is prohibited by law from extending financial assistanceto establishments relocating from one area to another See comment 12. and has internal review procedures that diminish the possibility of funding jobs relocating from outside the region (interregional shifts). In other words, EDA has procedures that would disqualify an employer relocating from Newark, NJ to a rural area in Montana. Further, interviewees in the May ,1997 report indicated that capital facilities assistance would not be a sufficient drawing card to cause an established business to uproot and relocate to an economically depressedrural or urban area within a region (intraregional shifts). See comment 13. The cost of job creation per total job created was estimated at $1,115 per job via the original regression analysis (May 1998 report). In the May 1997 report, if the cost per direct job ($3,058) is divided by 1.5 (the multiplier established to produce total jobs in the May 1998 report), the cost per total job created is about $2,000. GAO’s estimate indicates an EDA cost per job of about $1,600 for the average amount of an EDA grant. Thus, three different estimating procedures produced a job cost ranging from $1,100 to $2,000. This is a tight range of findings on job costs for these types of analyses. The Strength of the Combined EDA and GAO Analyses In its quest for answers, the GAO analyks has caused the research team to test more controlling variables than were included in the original analysis. This procedure when con-eczly done has reaffirmed the original results and continued to show that EDA funding has a positive and statistically significant relationship to job growth in a county. EDA cost per job in the average county, reflecting the original EDA and GAO analyses, falls within a range of $1,100 to S2,OOO.In the revised model, the cost of job creation in now controlled-for, smalZer and higher- See comment 14. wage areas has risen to about $11,000 per job. What does this mean? EDA job creation costs in smaller and higher-wage counties are higher than they are in larger counties having wage rates that are much lower. EDA has a positive and statistically significant effect in the smaller 4 18 GAOIRCED-99-1lR EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II counties,but it may cost EDA more to create this effect. Appendix A shows differing costsfrom different model specifications. Conclusion This researchproject representsone of the first times-if not the first time-that a federal agency has commissionedan independent evaluation of the impact of its programs.The size and nature of the programs make it difftcult to determinewith certainty the actual impact of relatively _.. small individual projects included within them. The statistical methods available are less than totally precise tools, depending as they do on the assumptionsand variables they employ. As See commertt 15. such, within the academiccommunity there is someroom to disagree.The relevant question here is not whether there exists a regression equation that can generatea statistically insignificant coeffkient on the level of EDA spending per job; surely an analyst can find such an equation. The real question is whether the preponderanceof the empirical evidencesuggeststhat the EDA Public Works Program is associatedwith real job creation. Given the fact that virtually all. of the analyses conducted in the EDA study indicate that the program does foster job retention and creation in distressedareas,it seemsthat the answerto this questionmust be “yes.” References Fisher. F. 1968a “Embodied Technology and the Existence of Labour and OUQW Aggregates,” Review ofkonomic Studies. Vol. 35, pp. 391-412. l%Bb. “Embodied Technology and the Aggregation of Fixed and Movable Capital Goods.” Review of Economic sIydips.VoI. 35. pp. 417428. A 1969. ‘The Existence of Aggregate Production Functions,* Econometrica. Vol. 37. No. 4. pp. 553-577. Haughwou~ A. 1998. -Aggregate Production Functions, lnterregionai Equilibrium, and the Measurement of Inframvcture Productivity.“Joumul of Urban Economics, Vol. 44. No. 2, pp. 216-227. (’ c 5 19 GAO/WED-99-11R EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II Appendix A-EDA Costs Per Job Under Various Treatments of County-level Effects The following table summarizes the results of the various analyses of county labor markets. Column 1 describes the controls for county-level effects included in the analysis; Column 2 provides the estimation method. With regard to the latter, the methods are either: (I) reduced form least squares with dummy variables (LSDV: for the employment equation only); (2) simultaneous equations (ZSLS: for both employment and compensation equations); or (3) generalized least squares (GLS: random effects estimation on the employment equation only). Column 3 reports the cost per job estimate retrieved from the estimation, with asterisks indicating the degree of confidence that the coefficient is distinguishable from zero in a statistical sense. Column l-Effects Included Column Z-Estimation Column 3-Point Method Estimate 1. Urban status, house prices, Generalized least squares S172*** demographic traits, recent labor W-9 market trends, and county random Reduced Form effects 2. Urban status, house prices, Least squareswith dummy %1,1X** demographic traits, recent labor variables (LSDV) Reduced market trends, and state fixed Form effects 3. Urban status, house prices, Least squareswith dummy s9 ,953**lv demographic traits, recent labor variabies (LSDV) Reduced market trends, state fixed effects, Form See comment 16. and lagged county employment 4. Urban status. house m-ices, I Two-stage least sauares $11.166** demographid traits, ;ecent.Iabor (2SLS) Simultaneous market trends, state fixed effects Equations lagged county employment (in employment equation), and compensation per employee (in compensation equation) ** Estimated effect is statistically distinguishablefrom zero with at least95% confidence. *** Estimated effect is statisticaIly distinguishablefrom zero witb at least99% confidence. Y The cost per job (and its associatedstandarderror) is identical when calculatedfrom a specification in which the dependentvariable is expressedas~XWZJIin employment since 1989. 6 20 GAO/R&ED-99-1lR EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II ’ Appendix B-Research’ Team Qualifications Research Organizations / Team Members Researchfor the Economic Development Administration was undertaken by Rutgers University, Center for Urban Policy Research (CUPR); New Jersey Institute of Technology, National Center for Transportation and Industrial Productivity (NJIT); Columbia University, Na- tional Center for Infrastructure Studies (NCIS); Princeton University, Woodrow Wilson School; the National Association of Regional Councils’ Economic Development and Planning Division (NARC); the University of Cincinnati, School of Planning and Urban Policy; and Economic Modeling Specialists, Inc. (EMSI). Rutgers University - Cgterfor Urban Policy Research (CUPR) For nearly three decades,the Center for Urban Policy Research has conducted a broad spectrum of urban research.In particular, CUPR has concentrated its efforts in analysis of infrastructure, public finance, economic impacts and forecasting, land use, environmental policy, and geographic information systems. The Center for Urban Policy Research has undertaken economic impact and infrastructure studies for the National Academy of Science, National Trust for Historic Preservation, Environmental Protection Agency, New York Metropolitan Transportation Commission, States of South Carolina and New Jersey, Southeast Michigan Council of Governments, and North JerseyTransportation Planning Authority. New Jersey Institute of Technology (NJIT) - National Centerfor Transportation and Industrial Productivity The National Center for Transportation and Industrial Productivity represents a substantial investment of the NJIT’s resources and research capacity in activities that are intended to address problems of relevance to local governments, the state, and the nation. The National Center’s researchinvolves federal and state transportation studies for motor vehicles and transit-based systems. Current researchprojects include estimation of multi-modal freight flows in the United States; smart sensorsfor freight movement; rail intermodal service planning; pipeline infrastructure studies to evaluate and develop criteria for the siting of natural gas and hazardous liquid transmission pipelines in proximity to the public in urban areas and in sensitive environments; economic and land use impacts of transportation projects; design and construction of prototype noise barriers; and seismic retrofitting of major bridges. Columbia University - National Centerfor Infrastructure Studies The National Center for Infiastntcture Studies was established to research technologies, techniques, and materials to improve the productivity and durability of infrastructure facilities in 7 21 GAOAXED-99-11R EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II urban areas. The Center has performed studies of infrastructure demand and supply with funding from federal agencies, states,and major cities. The Center has established a preventive maintenance management plan for the bridges of New York. developed environmentally responsible guidelines for New York City Bridges, and performed extensive destructive and non-destructive testing on many of the nation’s suspension bridges. It has recently developed an innovative concrete mixture substituting ground waste Princeton University - The Woodrow Wilson School of Public and International Affairs The Woodrow Wilson School of Public and International Affairs has more than fifty regular faculty members,most of whom have joint appointments with the Departments of Economics, Politics, or Sociology. The Woodrow Wilson School has research programs in demography, development, domestic policy, international studies, and survey research. The principal research: units are the Center of Domestic and Comparative Policy Studies, the Center of International ’ Studies, the Office of Population Research, and the Survey Center. The Center for Domestic and Comparative Policy Studies has undertaken multiple studies of the economic impacts of public works projects. National Association of Regional Councils (NARC) The National Association of Regional Councils promotes and encourages intergovernmental cooperation, recognition of the region as an economic entity, and cooperation among the nation’s public, private, and civic sectors. Researchthrusts include the capacity and ability of localities to , undertake economic development. University of Cincinnati - School of Planning and Urban Policy In the last twenty years, the fact&y of the SchooI of Planning and Urban PoIicy has conducted researchon community health, computer simulation and GIS, edge cities/metro-towns, environmental managementand policy, housing, inner-city development, international urban development, and urban design. Economic Modeling Specialists. Inc. (EMSI) EMS1 is a consulting firm specializing in regional economic modeling and analysis. EMS1 has constructed semi-survey economic models in a variety of settings from small rural communities to large and interconnected multistate regions. EMS1 has analyzed issues pertaining to energy and natural resource policy, transportation policy, fiscal impacts, fum siting, and a wide variety of issues pertaining to regional economic development and land-management planning. EMSI’s clients have included the Statesof Hawaii, Utah, and Idaho, the U.S. Forest Service, the U.S. Department of the Interior, an assortment of county and city governments, and private firms. 22 GAO/RCED-99-11R EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II Robert W. Burchell, Ph.D. (Rutgers University), Dr. Burchell has served as principal or co- principal investigator on ‘more than 60 research contracts in a thirty-year career at Rutgers University. He has conducted studies for the Federal Transit Administration, U.S. Department of Agriculture, Fannie Mae, U.S. Department of Housing and Urban Development, and other feder- al, state, and local agencies.For the last five years, his work has been concentrated in the areas of economic impacts and costs of infrastructure development. Louis J. Pignataro, D.Sc. (New Jersey Institute of Technology). Dr. Pignataro is Executive Director of NJIT’s Institute for Transportation and Distinguished Research Professor of Transportation Engineering. He has served as primary investigator for more than 35 sponsored researchprojects in a variety of areas,including pipeline infrastructure studies in the New York metropolitan area. F.H. (Bud) Gr@, Ph.D. (Columbia Universityl. Dr. Griffis has more than 37 years of experience in design, construction and maintenance of national and international infrastructure systemssuch as program management of the JFK International Airport redevelopment program, managementof the design and construction bf Ramon Airbase in Israel, and numerous infrastructure design and construction projects in Europe and the Far East. Andrew F. Haughwour, Ph.D. (Princeton Universiq). Dr. Haughwout is Assistant Professor of Public and International Affairs at the.Woodrow Wilson School and Faculty Associate, Offrce of Population Research,atPrinceton University. Professor Haughwout has written about the effects of city taxes on fiscal stability, the accumulation of assetsand liabilities by state and local governments, and the impacts of infrastructure investments on firms and households. His recent work appearsin such publications as Regional Science and Urban Economics. Journal of Urban Economics, and the National Tax Association ‘s Papers and Proceedings. His work is at the leading edge of his field and complements nicely with standard input-output analysis. John W.Epling, D.P.A. (National Association of Regional Councils). Dr. Epling brings to the project more than 30 years of experience working for local. regional, and state governments in four different stateson issues of economic development, infhastructure investment, urban and rural revitalization, and other areas.As the Executive Director of the National Association of Regional Councils, he has interacted with elected and appointed officials across the country on community and regional development and infrastructure needs. M. Henry Robison, Ph.D. (Economic Modeling Specialists, Inc.). Dr. Robison has twenty years of experience and numerous significant publications in the field of regional economic impact modeling and analysis. He is recognized for theoretical work blending regional input-output and spatial trade theory, and for development of community-level input-output modeling and analysis. He served for ten years as a faculty member and consultant to the University of Idaho, producing a wide array of grants and contract research. He is presently the Senior Research Economist at the Center for Business Development and Research, University of Idaho, and the Principal ResearchScientist for EMSI. 9 23 GAOLRCED-99-11R EDA’s May 1998 Final Report ENCLOSURE II ENCLOSURE II The following are GAO’s comments on the Department of Commerce’s letter dated February 12,1999. GAO’s Comments 1. It was not our purpose to create another statistical model but rather to evaluate the analysis and results contained in EDA’s May 1998.fina.l report. As part of our evaluation, we tested alternative specifications of the model used in the EDA report and found that for some of the specifications we tested, EDA’s grants had no effect on county employment. As a result, we believe that the conclusion in the EDA report that EDA’s grants create employment is unwarranted. 2. As we noted in our report, the statistical analysis in the May 1998 report does not determine whether the jobs attributed to EDA grants are newly created or relocated; it attempts to demonstrate that, holding constant other influences, such as whether the county is in a metropolitan area, counties that receive EDA grants have more jobs than counties that do not receive the grants. We continue to believe that the May 1998 report overstates the regression results with statements such as “EDA public works invesiments do, in fact, create jobs.” While assurances from applicants and certifications from businesses benefiting from EDA’s assistance that businesses will not relocate jobs may reflect that these businesses do not plan to move jobs from one location to another, the assurances do not account for market impacts that such moves might create. For example, if an EDA grant leads to increased jobs in a particular industry in one county, then the increased output in that industry may lead to lower prices, which, in turn, may result in reduced output and employment elsewhere. 3. See comment 1. 4. We continue to believe that to fully evaluate the impact of EDA grants on employment, it is important to consider the impact of EDA grants on employment using the results from a broader range of specifications. In response to our report, the researchers have analyzed additional specifications of their model. 5. We agree that including prior compensation levels is a good idea, and EDA’s comments indicated that its researchers conducted additional analyses that included both employment and compensation levels that were not presented in EDA’s May 1998 report. We did not include compensation levels in our analyses because the May 1998 report showed few effects attributable to compensation levels. After reviewing EDA’s comments, we investigated a number of specifications that take into account prior pay as well as prior employment, and we found that in some specifications EDA grants were significantly related to employment, while in others they were not. Whether or not EDA grants were significant depended on which prior year was used and how growth variables were modeled. This confirms our view that the results of EDA’s study were highly dependent on the model specification used and that the study’s definitive conclusions were unwarranted. Furthermore, we continue to believe that it is important to consider the impact of EDA grants on employment using the results from a broader range of specifications. 6. EDA’s comments indicate that the research team agrees with us that taking county- level effects into consideration is important. However, we do not believe that use of 24 GAO/ICED-99-11R EDA’s May 1998 Final Report ENCLOSURE II ENCLOSUREII “random effects estimation” is appropriate because it does not control for the bias Lintroduced by omitting unmeasured effects that are related to measured effects, such as omitting county size, which is related to EDA’s expenditure. 7. Although EDA’s contractors used other methods in addition to statistical analyses, in the 1998 report they wrote that the results of their statistical analysis suggest that EDA grants significantly influence employment. Our review focused on the regression analyses in the May 1998 report, in large part because regression analysis is a widely accepted methodology for exploring complex cause and effect relationships such as this one. We did not intend to create our own alternative model nor to critique ah the methods used by the research team. Our assessment involved determining whether the results of the analyses reported were sufficient to justify the team’s finding of a significant effect. For reasons discussed elsewhere (see comment 5), we continue to believe that the researchers’ analyses are not sufficient to reach the conclusion that EDA grants significantly, influence employment. 8. We believe that the researchers’ use of 1989 data could be problematic because the 203 projects being analyzed received their final payment by 1990. Thus, many would have been complete, or nearly complete, by 1989 and the projects’ impacts on wages, if any, could already be represented in the wage data from that year. We focused on 1985 and 1986 data in our analyses because most of the projects were approved in 1985 or later, so it is unlikely that the projects had any measurable impact on population or employment in 1986 or before. 9. See comment 5. 10. Although the researchers obtained this result in one specification, as discussed in comment 5, we analyzed a variety of specifications and found mixed results. 11. See comment 1. 12. See comment 2. 13. We did not intend for readers to interpret the figure of $1,600 per job created as a GAO estimate. Our intent was to show that alternative ways of dealing with certain technical issues would yield a range of estimates of the cost per job created. The estimated cost per job created depends on both the estimated statistical relationship between EDA spending and employment and the method used to derive cost estimates from that statistical relationship. Our analysis raises questions about both and suggests that substantial uncertainly remains about the cost per job created. This uncertainty extends beyond the range of $1,100 to $2,000 per job created 14. See comment 13. EDA’s revised model results in an estimate ofthe cost per job that is 10 times higher than its previous estimate of $1,100, further demonstrating the uncertainty present in its estimates of the cost per job created. In addition, according to EDA’s researchers, their revised model represents the cost of job creation in all counties, not in smaller and higher wage areas as stated in the comments. 15. We agree with EDA’s characterization that the real question is whether the preponderance of empirical evidence suggests that the EDA public works program is associated with job creation. However, the scope of our review was limited to the 25 GAOLRCED-99-1lR EDA’s May 1998 Find Report ENCLOSURE II ENCLOSURE II regression analyses in EDA’s May 1998 report. In our judgment, the results contained in that report do not provide sufficient evidence to conclude that EDA grants create jobs nor to reliably estimate the cost per job created. 16. According to correspondence with EDA’s contractors subsequent to EDA’s letter, “recent labor market trends” were not included in analyses 3 and 4 as indicated in column 1 of the table. (385770) 26 GAOIRCED-99-1lR EDA’s May 1998 Find Report Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and Mastercard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by tailing (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with “info” in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http&vww.gao.gov United States Bulk l&ate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GlOO Official Business Penalty for Private Use $300 Address Correction Reauested
Economic Development: Observations Regarding the Economic Development Administration's May 1998 Final Report on Its Public Works Program
Published by the Government Accountability Office on 1999-03-23.
Below is a raw (and likely hideous) rendition of the original report. (PDF)