United States General Accounting Office GAO Report to the Chairman, Subcommittee on Oversight and Investigations, Committee on Commerce, House of Representatives September 1997 AIR POLLUTION Limitations of EPA’s Motor Vehicle Emissions Model and Plans to Address Them GAO/RCED-97-210 United States GAO General Accounting Office Washington, D.C. 20548 Resources, Community, and Economic Development Division B-277440 September 15, 1997 The Honorable Joe Barton Chairman, Subcommittee on Oversight and Investigations Committee on Commerce House of Representatives Dear Mr. Chairman: To reduce and control air pollution, the Clean Air Act requires that the Environmental Protection Agency (EPA) establish national ambient air quality standards and that the states develop strategies for reaching and maintaining these standards. In order to evaluate the states’ strategies for reducing the atmospheric concentrations of three pollutants emitted by motor vehicles—carbon monoxide, hydrocarbons, and nitrogen oxides1 —EPA has developed a series of complex computer models, known as the MOBILE series, to estimate motor vehicle emissions. The current version is MOBILE5a. Both EPA and the states rely on this model to estimate future emissions after various control strategies are employed. Consequently, to the extent that the model erroneously estimates emissions, EPA could approve control strategies that may not be sufficient to attain the air quality standards, or, conversely, EPA could require that the states implement additional, often expensive control measures that may go beyond what is needed.2 In view of the pivotal role that the MOBILE model plays in the states’ efforts to improve air quality, you asked us to describe the major limitations in EPA’s MOBILE model and EPA’s process for improving the current and future versions of the model. EPA and a group of stakeholders have identified 14 major limitations in the Results in Brief current MOBILE model.3 Some vehicle emissions-producing activities are not accounted for in the current model, and other emissions-producing activities may not be adequately represented on the basis of the most 1 Carbon monoxide (CO) impairs lung and heart functioning, and hydrocarbons (HC) and nitrogen oxides (NOx) combine in sunlight to form ozone, or smog, which has been linked to a variety of health problems, ranging from eye, nose, and throat irritation to bronchitis, emphysema, and other serious lung diseases. 2 The states must obtain sufficient reductions in emissions or risk Clean Air Act sanctions that could result in losing millions of dollars in federal highway funds and jeopardizing industrial growth. Motor vehicles are one of several categories of sources contributing to air pollution. 3 EPA and a group of stakeholders working with the agency on modeling issues identified a number of limitations in MOBILE5a and selected 14 of these for potential revision in MOBILE6. Page 1 GAO/RCED-97-210 Air Pollution B-277440 recent information. According to EPA, much of this information has become available since MOBILE5a was released. These limitations cause the model to underestimate vehicle emissions in some cases and overestimate them in others. For example, the impact of road grade—such as when a car climbs a hill—is not a part of the current model, although some studies have indicated that both the increased load on the engine from climbing a hill and the decreased load that accompanies engine deceleration significantly increase vehicle emissions. Similarly, other studies indicate that some activities—such as when someone drives at high speeds, accelerates quickly, runs the air conditioner, or uses a higher mileage vehicle4 —are inadequately represented in the model. For example, some testing indicates that nitrogen oxide emissions from running the air conditioner may be from 30 to 75 percent greater than estimated by the MOBILE model. Conversely, another study indicates that carbon monoxide and hydrocarbon emissions from higher mileage vehicles may be significantly less than the model’s estimates. EPA plans to address most of these limitations in its next revision to the MOBILE model; however, according to agency officials, three of the limitations, including the impact of road grade, will probably not be addressed until later because of a combination of factors. According to agency officials, these include the negligible impact on emissions inventory predictions, a relatively low priority ascribed by EPA and stakeholders, the cost and length of time required for these studies relative to the schedule for release of MOBILE6, and the emergence of new technologies that will make the improvements more feasible or cost effective in a few years. EPA officials pointed out that they have updated the estimating capabilities of the MOBILE model 10 times since it was first introduced in 1978. Irrespective of these limitations, there are specific actions—most of which were recommended by the Science Advisory Board in its 1989 resolution5—that, when followed, can enhance a model’s estimating capabilities. Among other things, these actions involve documenting the implicit and explicit assumptions that are the basis of the formulas contained in the model, obtaining external stakeholders’ input during the model’s development, and having the model peer reviewed before it is used. EPA officials acknowledged that, primarily because of resource limitations, until recently such actions have been delayed or forgone. 4 EPA’s current model assumes that the emissions systems of vehicles with 50,000 or more odometer miles deteriorate much more rapidly than the systems of other vehicles. 5 EPA’s Science Advisory Board recommended many of these processes in its January 1989 resolution on models, including one process—peer review—that has been required by EPA policy since January 1993. The Board is a legislatively established body of independent experts who provide advice to the EPA Administrator on scientific and engineering issues. Page 2 GAO/RCED-97-210 Air Pollution B-277440 However, EPA is developing the next model, MOBILE6, with significantly increased openness and input from other stakeholders. EPA also plans to carry out the actions recommended by the Science Advisory Board, such as peer review, as part of its program for developing MOBILE6, due to be issued in late 1998. The current MOBILE model, MOBILE5a,6 also known as the EPA mobile source Background emissions factor model, is a computer program7 that estimates the emissions of carbon monoxide, hydrocarbons, and nitrogen oxides for eight different types of gasoline-fueled and diesel highway motor vehicles.8 The model consists of an integrated collection of mathematical equations and assumptions about the emissions from vehicles manufactured from 1960 to 2020; generally, the cars produced in the 25 most recent model years are assumed to be in operation in any given calendar year.9 The first MOBILE model was made available for use in 1978; since that time, major updates and improvements to the model have been made as more has become known about the complexity of the factors affecting vehicle emissions, as measurement devices have improved, and as more data have been collected. According to agency officials, these improvements have resulted in the refinement of emissions estimates for evaporative emissions (such as occur when the fuel tank and fuel system heat up on a hot summer day); for the uncorrected in-use deterioration (wear and tear) that results from poor vehicle maintenance or tampering; and for other factors. In its simplest form, EPA’s MOBILE model allows the model user to produce a number—an estimated quantity of emissions for the three pollutants of concern—by multiplying the estimated emissions per mile for an average urban trip times the estimated number of trip miles traveled in an area. Over the years, however, researchers have learned that vehicle emissions 6 MOBILE5a, issued in March 1993, is the current version of the EPA MOBILE model that the states are required to use; EPA issued an update in September 1996, known as MOBILE5b, as an option for the states to use in calculating selected emissions reduction credits. Unless otherwise indicated, the limitations discussed in this report relate to both MOBILE5a and MOBILE5b. 7 To facilitate computer programming, EPA created two other models—one for fuels, another for inspection and maintenance measures—whose outputs are inputs to MOBILE5a. Unless otherwise specified, references to the MOBILE model hereinafter in this report will include these input models. 8 The eight classes of vehicles are light duty gasoline-powered vehicles (passenger cars); light duty gasoline-powered trucks up to 6,000 pounds gross weight, and from 6,000 to 8,500 pounds gross weight; heavy duty gasoline-powered trucks; light duty diesel vehicles; light duty diesel trucks; heavy duty diesel trucks; and motorcycles. 9 Vehicles more than 25 years old are lumped into one category called “25 years and older.” Page 3 GAO/RCED-97-210 Air Pollution B-277440 are highly complex. For example, EPA and others have indications today that as much as half of all hydrocarbon emissions from motor vehicles are evaporative emissions, under certain conditions. To compensate for the complexities of these and other emissions-producing activities, EPA has periodically adjusted its basic formula—through the use of revised “correction factors”—to approximate vehicle exhaust emissions in a range of situations. In essence, the correction factor is a multiplier added to the basic formula (miles traveled times emissions rate per mile) to adjust the model’s output to more closely reflect actual emissions. Except for California,10 EPA supplies the baseline emissions rates and correction factors for other model users—primarily state and local agencies—that typically supply their own estimates of the number of vehicle miles traveled, according to agency officials, as well as many other local area parameters, such as the average ambient temperature, vehicle classifications, and types of fuels sold. The MOBILE model exists because precise information about the emissions behavior of the approximately 200 million vehicles in use in the United States is not known, yet the need exists to estimate the impact of motor vehicles on air quality. For the states, the MOBILE model is a tool for constructing emissions inventories,11 creating control strategies, producing state implementation plans (SIP),12 and—subsequently— demonstrating control strategy effectiveness to EPA and others. For example, the states are allowed to vary a number of control strategy features, including the types of fuels used, the type of inspection and maintenance (I&M) testing network,13 the frequency of I&M testing, the ages and types of vehicles to be inspected, the stringency of the tailpipe test, the number and percent of inspected vehicles that may receive a waiver,14 and a host of other factors. The states may choose among a number of 10 With EPA’s approval, California uses its own emissions factor model to estimate vehicle emissions. 11 National emissions are estimated annually by EPA on the basis of statistical information about each of more than 450 source categories, including the emissions factors and control efficiencies for each source category. These aggregated emissions show emissions trends at national and regional levels for man-made pollutants. 12 These plans explain in detail how, and by how much, state and local areas will reduce or control emissions from both mobile and stationary sources. 13 Testing networks may be test-only—where inspectors are prohibited from making repairs; test-and-repair—where inspectors/mechanics are allowed to make repairs; or hybrid—where both types of facilities are allowed. 14 The act requires that vehicle owners spend at least $450 toward emissions-related repairs before a waiver may be granted. Page 4 GAO/RCED-97-210 Air Pollution B-277440 control options as long as the state’s control strategy achieves at least as many reductions as required by the Clean Air Act.15 For EPA, the MOBILE model is a tool for evaluating the adequacy of a state’s emissions inventory estimate, motor vehicle control strategies, and implementation plans. In essence, the model’s estimates provide EPA regulators with critical information that is used to evaluate the adequacy of a state’s program and the relative benefits of various policies to control motor vehicle emissions. Additionally, the model’s estimates can affect state policy decisions on issues such as the content and volatility of fuels, and some decisions on highway improvement projects. For example, the 1991 Intermodal Surface Transportation Efficiency Act (ISTEA) required, among other things, that state transportation improvement programs in certain nonattainment areas16 conform with the applicable state implementation plan developed under the Clean Air Act. Although the model’s original purpose was to support the development of mobile source emissions inventories, over the years its role and influence have been expanded considerably. Today, its estimates have a substantial influence not only on state and local programs but also on the automobile and oil industries, environmental and trade organizations, the public, and others. According to estimates derived partly from the MOBILE model, motor vehicles produce about 90 percent of the carbon monoxide, 50 percent of the hydrocarbons, and 30 percent of the nitrogen oxides emitted annually in major urban areas. EPA officials are examining 14 areas in the current MOBILE model in which Limitations in the major limitations exist. According to agency officials, it is their plan for Current Model each new version of the MOBILE model to reflect the most recent testing, data collection, and research that are available. They pointed out that EPA has updated the estimating capabilities of its MOBILE source emissions model 10 times since the model was first introduced in 1978. Table 1 briefly summarizes the areas in which major limitations exist, as well as EPA’s plans to address these limitations in its next revision to the model, MOBILE6, due to be issued in late 1998. (Additional information on these 14 areas is provided in app. I.) 15 The states must achieve percentage reductions in HC and, for CO and NOx, must submit plans for attaining the national ambient air quality standards by the dates specified in the act relative to the seriousness of their pollution problems. 16 Areas that have not yet reached attainment with the national ambient air quality standards are classified by EPA as nonattainment areas, generally ranging in seriousness from marginal to extreme. The more serious the classification, the more stringent the control measures. ISTEA refers specifically to areas that are in nonattainment for ozone or carbon monoxide. Page 5 GAO/RCED-97-210 Air Pollution B-277440 Table 1: Summary of Major Limitations in the MOBILE5a Model Planned to be addressed in Area in MOBILE model Brief description of limitationa MOBILE6 1. Emission estimates for Higher speeds produce increasingly larger emissions of CO, HC, and NOx; Yes higher speeds, especially technological limits of previous dynamometersc meant most data were collected b speeds in excess of 65 mph. at speeds below 57 mph; increases in speeds above 65 mph have left data gaps in current model that are not adequately represented by existing correction factors; ongoing work will be used to revise these estimates. 2. Representation of Rapid acceleration and deceleration can produce large increases in emissions Yes emissions from rapid of CO and HC; technological limits of previous dynamometers meant most data acceleration and were collected at or below acceleration rates of 3.3 mph/second; one EPA deceleration, including study indicated that about one-third of trips had acceleration rates of more than aggressive driving 7 mph/second; concerns exist as to whether these increased emissions are behaviors.b adequately represented by existing correction factors; ongoing work will be used to revise these estimates. 3. Representation of Until vehicle engines reach normal operating temperatures, emissions typically Yes emissions immediately after bypass emissions control equipment for 3 to 5 minutes, making cold starts one engine start-up, known as of the single largest emissions-producing activities; recent studies indicate that cold start emissions.b cold starts occur more frequently than estimated in the Federal Test Procedure, thus increasing emissions; however, newer technology vehicles now reach normal operating temperatures more quickly, thereby decreasing emissions; concerns exist as to whether these activities are adequately represented by the current MOBILE model and whether they should be separately calculated on the basis of activity levels as opposed to representing one portion of EPA’s typical driving cycle; EPA has already announced plans to separate cold starts from other driving cycle activities. 4. Representation of Air conditioners place an additional load on a vehicle’s engine and cause a Yes emissions from air conditioner vehicle to produce more NOx emissions than when the air conditioner is not usage.b used; although EPA had added 10 percent more load to the dynamometer to simulate the effects of air conditioners on emissions, testing on a few vehicles indicates that the emissions of NOx may be from 30 to 75 percent greater at some speeds than currently represented by the MOBILE modele; the results of recent air conditioner emissions studies are planned to be reflected in the model’s next revisions. 5. Representation of Increased road grade can result in substantial emissions increases, especially No emissions from road grade, in CO and NOx; due to the difficulty of testing for this driving activity and the such as when a car climbs a difficulty of plotting road grades for millions of miles of highways, this activity hill.b has not been represented in EPA’s model to date; due to the expense and time required for road grade testing and mapping, EPA plans to add road grade emissions estimates in several years when it develops MOBILE7. 6. Representation of high High emitting vehicles account for a disproportionate amount of vehicle Yes emitting vehiclesd in the emissions; thus, the estimate of their relative proportion in the overall vehicle MOBILE model’s supporting population is critical to accurate inventory emissions estimations; a California database. study found high emitting vehicles underrepresented in its own model by nearly 5 times; concerns exist as to whether these high emitting vehicles are appropriately represented in the current EPA model; ongoing I&M lane testing in three states will be used to review these estimates. (continued) Page 6 GAO/RCED-97-210 Air Pollution B-277440 Planned to be addressed in Area in MOBILE model Brief description of limitationa MOBILE6 7. Representation of Lower volatility fuels produce less emissions, especially HC and CO emissions; Yes emissions from lower over the years, low volatility fuels have been produced that are lower than the polluting fuels, especially lowest level EPA estimated in 1989 when it placed a lower limit restriction in the fuels with lower volatility. MOBILE model, below which low volatility fuels received no additional emissions reductions credits; additionally, one study found that the model overestimates the CO emissions benefits of oxygenated fuels; EPA plans to eliminate the lower limit restriction, review correction factors for lower volatility fuels, and review CO estimates for oxygenated fuels. 8. Representation of On the basis of a small sample, EPA added new correction factors in 1992 on Yes emissions system the assumption that emissions control systems perform much worse after deterioration for vehicles with 50,000 miles; some testing indicates that this adjustment now causes the 50,000 or more odometer MOBILE model to overestimate the emissions from fuel-injected vehicles with miles. 50,000 or more odometer miles.; EPA plans to lower this adjustment; ongoing work will determine the extent of change. 9. Emissions estimates and Vehicle condition is one of the most important factors contributing to excess Yes assumptions for vehicle emissions; more recent and more complete data are needed about many facets inspection and maintenance of I&M programs, including the impact of newer on-board computer technology, (I&M) programs. better emissions control system warranties, the durability of equipment, and alternative ways to complement inspection programs and improve repair effectiveness; for MOBILE6, EPA plans to add an I&M flexibility module covering most of these areas so that individual states’ studies and ongoing EPA studies of I&M testing in three states can be used to revise these estimates as study data become available. 10. Estimates and Hydrocarbons evaporate from the fuel tank and fuel lines when the vehicle is Yes assumptions for non-tailpipe parked; improved testing over a longer time period with more realistic evaporative emissions when temperature increases is ongoing to better reflect actual emissions and refine the vehicle is not operating. existing correction factors in the MOBILE model; however, some vehicles that leak fuel are now believed by EPA to be so significant that these will become a separate emissions category in MOBILE6; indications are that vehicles with such fuel leaks—while a small percentage of the fleet—can exceed the evaporative emissions of corresponding non-leaking vehicles by one to two orders of magnitude. 11. Emissions estimates and EPA’s supporting data on the in-use emissions of this category of vehicles are No assumptions for the about 20 years old; certification standards are higher for these vehicles than inspection and maintenance their light duty counterparts, and they are generally older and driven more miles (I&M) of heavy duty annually than their light duty counterparts, although improved emissions vehicles—those with a gross technology has lessened the contributions of individual vehicles; some studies vehicle weight of 8,501 are under way, but agency officials question whether sufficient I&M data will be pounds or more. obtained in time to change emissions estimates for heavy duty vehicles. 12. Data characterizing Many model assumptions, such as emissions control system deterioration, are Yes vehicle fleet. closely tied to accumulated mileage; the MOBILE model’s data on mileage accumulation has been superseded by more recent data indicating that passenger cars are driven about 2,000 miles, or 10 percent more, than currently estimated in the MOBILE model; also, heavy duty vehicles are believed to constitute a significantly larger percentage of the overall vehicle fleet than currently estimated in the MOBILE model; EPA plans to revise these estimates at the national level and notes that this is an area where model users often supply their own local data. (continued) Page 7 GAO/RCED-97-210 Air Pollution B-277440 Planned to be addressed in Area in MOBILE model Brief description of limitationa MOBILE6 13. Greater distinctions in Roadway classification (freeway, arterial, feeder/collector street, along with their Yes roadway classifications. associated level of service) has a substantial impact on emissions; the current model was designed for estimating areawide emissions over an entire trip and has not provided for assessing emissions’ impacts on a smaller scale, yet today’s users need a model that provides different estimates for different roadways; for MOBILE6, EPA plans to provide separate model emissions estimates for at least three roadway classifications. 14. Quantifying the Traditionally, when estimates are made from limited data, the magnitude of No uncertainty of the model’s uncertainty, or range, associated with the estimate is provided; however, this estimates. has not been done for any versions of the MOBILE model. Several studies suggest that the current model’s uncertainty may be large; however, many factors, including cost, will likely prevent EPA from reporting the uncertainty of its MOBILE6 estimates, although the agency plans to move in this direction. Note: Representatives from EPA and from the Federal Advisory Committee Act (FACA) workgroup are continuing to work on other less significant limitations in MOBILE5a that are not addressed above. From an original list of 90 items, an EPA-sponsored FACA subcommittee has identified 47 high-priority items for improvement in the current model; however, many of these items involved overlapping issues, and in some instances, two or more of these items have been consolidated to form the 14 categories of limitations discussed above. Additionally, some of these items deal with suggested procedural changes in the process that EPA should follow in making MOBILE model revisions. Procedural changes in EPA’s modeling process are discussed in the next section of this report. a In some instances, the suggested emissions impact is based on a few studies of vehicle activities that EPA is currently trying to corroborate. b While many model assumptions are interconnected, these limitations relate primarily to limitations in the Federal Test Procedure (FTP) for certifying new cars, which was revised in October 1996 to better estimate emissions from high speeds, aggressive driving, cold engine starts, and air conditioner use; these changes in the revised FTP have not been reflected in MOBILE5a (issued in March 1993), nor in MOBILE5b (an optional version that was issued September 1996); the relationship of FTP assumptions to EPA’s MOBILE model estimates is discussed more fully in appendix I. c A dynamometer is a treadmill-like device that simulates the load placed on a vehicle by electronically controlling inertial forces while the vehicle’s wheels are driven at various pre-determined speeds, including acceleration and deceleration. d High emitters are generally defined as vehicles with emissions more than twice the applicable standard for the particular make and model of vehicle, and are often described on the basis of the amount of excess emissions they produce. For CO and HC, EPA classifies such excess emitters as high, very high, and super emitters; for NOx, vehicles are classified as either normal or high. e These tests compared actual air conditioner emissions with the FTP’s estimate of such emissions by adding 10 percent additional load to the dynamometer; MOBILE5a employed essentially the same correction factor, thereby estimating air conditioner emissions in the same manner and amount as the FTP. While acknowledging that some vehicle emissions-producing activities are not accounted for in the current model and that other emissions-producing Page 8 GAO/RCED-97-210 Air Pollution B-277440 activities are not adequately represented in the current model on the basis of the most recent information, EPA officials said that it is important to note that EPA has conducted and/or partially funded some of the studies that have led to the new data that now question the old estimates and assumptions. Additionally, they said that EPA has work under way to address most of these limitations. For example, since its formation in September 1995, an EPA-sponsored Federal Advisory Committee Act (FACA) subcommittee workgroup has identified 47 high-priority items for improvement in the current model, and EPA and workgroup representatives are examining these limitations. EPA is also in the process of developing new procedures for improving models in general, which are discussed below. Several model experts told us that it is the nature of models such as EPA’s Process for MOBILE5a to have limitations and to be in a continuous improvement mode. Improving the Agency officials agreed with this assessment, noting that the current MOBILE Model model is better than any previous versions and reflects consistent growth in the quality and quantity of information available on very complex issues. Additionally, they pointed out that—through the FACA workgroup process—the revisions to the next MOBILE model, MOBILE6, have been undertaken with significantly increased openness and input from other government agencies, academia, the automobile and oil industries, environmental groups, and others. Our contacts with representatives of these groups confirmed this increased level of external stakeholders’ involvement in preparation for MOBILE6. Several commended EPA’s efforts in recent years to reach out to persons outside of the agency, and some noted that the outreach effort had given them a much greater appreciation of the model. While acknowledging that, historically, there have been few firm criteria on the processes that should be followed when creating or revising a model such as the MOBILE model, the executive director of EPA’s Science Advisory Board (SAB) told us that the agency has a project under way to develop agencywide procedures for improving models. According to the project director, the Office of Research and Development (ORD) is planning a workshop in December 1997 to discuss the status of modeling across several media and other modeling issues, including the need for better agencywide modeling procedures. The project director also said that the Science Advisory Board’s January 1989 resolution on models was one of the best documents available on the processes for creating and improving models in general. The SAB executive director and the ORD project director Page 9 GAO/RCED-97-210 Air Pollution B-277440 told us that in their opinion, there are specific actions—most of which were recommended by the SAB in its 1989 resolution—that, when followed, can enhance a model’s predictive capabilities. Among other things, these actions include the following: • Obtaining external stakeholders’ input to ensure that the model’s assumptions and formulas receive critical review by those not involved in the model’s development. • Documenting the implicit and explicit assumptions so that others can evaluate the basis of the formulas embedded in the model. • Performing sensitivity analyses over key parameters to identify the most sensitive parameters and to establish the areas most in need of further research. • Verifying the adequacy of the model’s mathematical code. • Testing the model’s predictions with laboratory and field data to confirm that the model generates results consistent with its underlying theory. • Conducting peer review to enhance the quality, credibility, and acceptability of the model’s applications. The SAB executive director pointed out that because of continuing concerns with the quasi-regulatory use of agency models, EPA issued agencywide guidance in 1994 specifically calling for the peer review of such models. This directive was a follow-on to EPA’s January 1993 agencywide policy requiring peer review of the scientific and technical work products used to support agency decisions. In September 199617 and March 1997,18 we reported and testified on the uneven implementation of EPA’s peer review policy, including that the MOBILE model had not been peer reviewed. EPA agreed with our recommendations for educating staff and managers about the merits of and procedures for conducting peer review and for ensuring that all relevant products are considered for peer review. The agency has set in motion a three-pronged approach to improve the implementation of peer review agencywide, including peer review of the next MOBILE model. According to the ORD project director, it is too early in EPA’s study to predict whether the agency may recommend that EPA require its offices to follow the other actions listed above when creating or revising models. According to Office of Mobile Sources (OMS) officials, they plan to carry out all six of the above activities as part of their improvement process for 17 Peer Review: EPA’s Implementation Remains Uneven (GAO/RCED-96-236, Sept. 24, 1996). 18 Peer Review: EPA’s Implementation Remains Uneven (GAO/T-RCED-97-95, Mar. 11, 1997), testimony before the House Subcommittee on Energy and Environment, Committee on Science. Page 10 GAO/RCED-97-210 Air Pollution B-277440 MOBILE6 and noted that some of these activities are already well under way, such as involving external stakeholders. For example, OMS held its first stakeholder meeting in June 1994, established a FACA mobile modeling workgroup in July 1995, and has held five meetings since that time to obtain external views by those not involved in the model’s development, according to agency officials. OMS officials acknowledged that some of the key formulas in the current model have not been properly documented,19 that full-scale sensitivity analyses have not been performed since May 1990 (when they were performed for MOBILE4.1),20 that fewer resources have resulted in fewer confirming data, and that the MOBILE model has not been peer reviewed. However, they said they have efforts under way or planned to address these and other modeling needs. For example, one of the recommendations of the FACA mobile modeling workgroup—made up of representatives from EPA, state and local agencies, industry, environmental groups, and academia—is that EPA more fully document the model’s assumptions. Additionally, OMS plans to perform sensitivity analyses for the next version of the model and to have the studies supporting key changes for MOBILE6 peer reviewed. Also, OMS officials explained that as changes are proposed for each area of major limitations in the model, they plan to have the entire area peer reviewed.21 Agency officials explained that declining modeling resources have affected the pace of model improvements over the years, particularly their ability to confirm the model’s estimates with large numbers of vehicle tests. For example, a study of the emissions characteristics of 100 passenger cars for both exhaust and evaporative emissions could cost from $1.4 to $1.6 million, according to the agency’s current estimates, and still not address the emissions impacts of road grade, air conditioning, or most fuel studies. Studies of heavy duty trucks and other larger vehicles would cost considerably more. As an illustration of the magnitude of the task compared with available resources, we obtained EPA’s estimate of mobile modeling needs in response to the mobile source requirements envisioned for the 1990 Clean Air Act amendments. This June 1990 analysis indicated that the Office of Mobile Sources would need about $60 million for the modeling improvements known at that time. Since then, because of higher 19 Reports by Sierra Research, Systems Applications International, and the Department of Transportation’s Federal Highway Administration, among others, found that some of the key formulas in the MOBILE model are not properly documented, making it difficult for researchers to know what data or assumptions were used in developing the formulas. 20 This was the last sensitivity analysis EPA published; agency officials indicated that limited sensitivity analyses were performed over selected parameters—such as key speeds, temperatures, and fuels—in MOBILE5 (Dec. 1992) and MOBILE5a (Mar. 1993) but that these analyses were limited and not published. 21 OMS officials explained that they are still early in the peer review process and have not yet developed a peer review plan for the MOBILE model. Page 11 GAO/RCED-97-210 Air Pollution B-277440 priority needs, OMS has been allocated only $21.8 million, cumulatively, for modeling improvements, although the research needs have increased. However, many more groups have become involved in non-EPA-funded vehicle emissions studies than in the past, allowing EPA to benefit from their studies and observations. Additionally, in some instances, researchers have sought EPA’s input on study protocols beforehand and have shared the data collected with EPA afterwards. While concerned about resources, OMS officials explained that making model improvements is an ongoing, continuous process—and one that will continue after MOBILE6 is issued in 1998. They pointed out that their goal is for each new version of the MOBILE model to reflect the latest testing, data collection, and research. While still not able to quantify the improvements, they said that in their opinion, each new version of the MOBILE model is better than its predecessor. We provided copies of a draft of this report to the Environmental Agency Comments Protection Agency for its review and comment. We obtained comments from EPA officials, including the Director of the EPA Office of Mobile Sources. EPA agreed with the overall message of the report but expressed concerns with imprecise language and suggested several changes to clarify information in the report. For example, EPA suggested that in lieu of describing some limitations associated with its use of the Federal Test Procedure as a basis for estimating emissions as “FTP assumptions,” since the FTP is a specific, codified test cycle, these would be better described as “FTP parameters.” EPA also suggested that we provide specific citations for four studies referred to in appendix I. We made the language changes suggested by EPA, including adding the citations. EPA was also concerned that the report did not clearly distinguish between limitations that may result in only trivial emissions impacts and those that could be significant. Where it was possible to do so, we believe the estimated emissions impact had already been quantified or qualitatively described in the report. Additionally, as noted in the section on the uncertainty limitation, one of the 14 limitations of the MOBILE model is that it does not currently have information about, nor estimates of, the uncertainty associated with its emissions estimates. However, we agree that researchers viewed some limitations as having a more significant impact on emissions than others, and we have provided this view in the report. Appendix II contains the agency’s overall written comments. Page 12 GAO/RCED-97-210 Air Pollution B-277440 We conducted our review from October 1996 through August 1997 in accordance with generally accepted government auditing standards. A detailed discussion of our scope and methodology is provided in appendix III. As arranged with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 15 days after the date of this letter. At that time, we will send copies to the Administrator of EPA and other interested parties. We will also make copies available to others upon request. Please call me at (202) 512-9692 if you or your staff have any questions. Major contributors to this report are listed in appendix IV. Sincerely yours, Lawrence J. Dyckman Associate Director, Environmental Protection Issues Page 13 GAO/RCED-97-210 Air Pollution Contents Letter 1 Appendix I 16 Major Limitations in MOBILE5a Appendix II 30 Comments From the Environmental Protection Agency Appendix III 32 Objectives, Scope, and Methodology Appendix IV 34 Major Contributors to This Report Table Table 1: Summary of Major Limitations in the MOBILE5a Model 6 Page 14 GAO/RCED-97-210 Air Pollution Contents Abbreviations API American Petroleum Institute AQIRP Air Quality Improvement Research Program CAA Clean Air Act CARB California Air Resources Board CO carbon monoxide CRC Coordinating Research Council EPA Environmental Protection Agency FACA Federal Advisory Committee Act FTP Federal Test Procedure GAO General Accounting Office HC hydrocarbons ISTEA Intermodal Surface Transportation Efficiency Act I&M inspection and maintenance OMS Office of Mobile Sources ORD Office of Research and Development NOx nitrogen oxides OBD on-board diagnostics RVP reid vapor pressure SAB Science Advisory Board SIP state implementation plan Page 15 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a According to agency officials and model experts we contacted, it is the nature of models to have limitations and to be in a continuous improvement mode. As a result, the Environmental Protection Agency (EPA) has periodically updated the estimating capabilities of its mobile source emissions model to reflect new information as data have become available; MOBILE5a reflects the 10th major revision since the model was first introduced in 1978. The following sections provide additional information on 14 areas in which major limitations exist in EPA’s current MOBILE model, MOBILE5a. Five Limitations Related to The underlying basis for EPA’s original model, and all subsequent versions, Use of the Federal Test has been the Federal Test Procedure (FTP),1 a laboratory dynamometer2 Procedure as a Basis for test used to certify new cars against new-car emissions standards. The FTP is roughly based on a typical urban area trip, complete with starts and Inventory Modeling stops, covering 7.5 miles in the Los Angeles urban area in the late 1960s. Such a trip is known as a driving cycle, which can be approximated on a dynamometer.3 Primarily because of the limitations in past dynamometers,4 the FTP driving cycle parameters stipulate, among other things, that vehicles average 19.6 miles per hour (mph) over the 7.5 mile trip, do not exceed 57 mph, accelerate gradually (not to exceed 3.3 mph/second), and travel on a flat surface. Additionally, EPA added 10 percent to the FTP dynamometer load in an attempt to simulate the effects of air conditioner usage. However, five of the major limitations in the current MOBILE model relate to FTP parameters. These five are (1) emissions from road grade, (2) emissions from air conditioner usage, (3) emissions at higher speeds, 1 Since 1991, EPA has used dynamometer test data from thousands of vehicles tested in IM240 inspection lanes in Hammond, Indiana, to improve the basic emissions rate estimates in the MOBILE model. The IM240 test is a 240-second test designed to replicate certain segments of the longer FTP test and, as such, has the same basic benefits and limitations as the FTP test. 2 A dynamometer is a treadmill-like device that simulates the load placed on a vehicle by electronically controlling inertial forces while the vehicle’s wheels are driven at various pre-determined speeds, including acceleration and deceleration. The new FTP requires improved single-roll dynamometers, 48 inches in diameter, with higher power absorption capacity to provide better replication of actual emissions (or a system that provides equivalent or superior results), replacing the smaller (about 9 inch) double-roll types. 3 EPA’s driving cycle, known as the EPA Urban Dynamometer Driving Schedule, tests a vehicle over several driving scenarios during a 1,877-second, or about a 31-minute dynamometer test. “Cold-start” emissions are represented by the first 505 seconds of emissions, after which so-called “hot stabilized” emissions are collected to represent the period when the vehicle is fully warmed up and both the engine and catalytic converter have reached typical operating temperatures (the 1,877-second driving cycle also includes a “hot start” component). 4 According to EPA, FTP driving cycle parameters, such as top speed and acceleration rate, until recently were substantially constrained from approximating actual driving conditions by the physical limitations in dynamometer technology. Page 16 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a (4) emissions from aggressive driving, and (5) emissions immediately after engine start-up (cold- start). Agency officials have long recognized that some of these original FTP parameters were not representative of actual driving conditions and, to compensate for these limitations, have added correction factors to the MOBILE model to estimate what emissions would be for speeds in excess of 57 mph, for rapid acceleration beyond 3.3 mph/second, and for other scenarios, such as different temperatures or different fuels. For example, the impact of temperature on emissions can be substantial. Consequently, while FTP testing has been performed between 68 and 86 degrees Fahrenheit, EPA’s MOBILE model used a correction factor to estimate that 1995 exhaust emissions of hydrocarbons would be 3 times greater at 25 degrees than at the FTP temperatures. However, MOBILE5a does not account for the impact of road grade—such as when a car climbs a hill—although some studies have indicated that both the increased load on the engine from climbing a hill and the decreased load that accompanies engine deceleration significantly increase vehicle emissions. According to agency officials, it is not expected that MOBILE6 will have adjustments for road grade, although such adjustments are being planned for MOBILE7. In addition to being uncertain about the amount of emissions related to road grade, EPA officials explained that obtaining the basic data from instrumented cars and chase cars to make such adjustments would be expensive at this time and that because of the cost and length of time required for these studies, the impact of road grade will probably not be addressed until MOBILE7. Additionally, an equally important consideration is that once these basic data on the effects of road grade on emissions are obtained, state and local agencies would have to plot road grades for millions of miles of roadways in their jurisdictions, also a costly and time-consuming activity. However, as global positioning technology5 for vehicles becomes less costly and more widely available and used, it is envisioned that the impacts of road grade emissions will be modeled in the future. According to EPA officials, the amount of data needed to estimate the impact of road grade is still years away. They also noted that time would be needed to develop consistent guidance on how state and local agencies should go about collecting these data. With respect to the representation of other emissions-producing activities not represented by the FTP, EPA officials have made periodic adjustments but recognize that the current model’s correction factor adjustments may not reflect the most up-to-date information. For example, as noted in the 5 Global positioning system and on-board vehicle computer technology allow a driver to electronically determine his location on earth, among other things. Appropriately instrumented vehicles would use these technologies to plot and record both grade and location. Page 17 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a EPA-FACA materials disseminated in March 1997, the 10-percent additional load intended to simulate the effect of air conditioner usage “is obsolete.” More recent information indicates that nitrogen oxide emissions may be from 30 to 75 percent greater at some speeds than the current model’s estimates when the air conditioner is used. Additionally, increases in speeds above 65 mph have left data gaps in the current model that are not adequately represented by existing correction factors. Others have recognized that the MOBILE model’s estimates are inextricably tied to the FTP’s parameters, and some studies have questioned the representativeness of key assumptions as they relate to the FTP. For example, the California Air Resources Board (CARB) commissioned Sierra Research in 1993 to develop an improved driving cycle—known as the Unified Driving Cycle—by using an instrumented “chase car” to better characterize typical urban driving patterns. Among other things, this driving cycle allows cars to travel up to 67.2 mph (versus 57 mph for the FTP), allows for acceleration at a rate of up to 6.9 mph/second (versus a maximum of 3.3 mph/second for the FTP), and uses an average speed of 24.6 mph (versus 19.2 for the FTP). Several model experts believe these parameters more closely approximate actual driving conditions today. According to a 1993 CARB study, the FTP may underestimate hydrocarbon, carbon monoxide, and nitrogen oxide emissions by 27, 68, and 17 percent, respectively. A 1993 study sponsored by EPA’s FTP improvement project found that more than one-third of the trips studied had acceleration rates of more than 7 mph/second—more than double the FTP’s maximum of 3.3 mph/second rate. Similarly, another 1993 EPA-sponsored study of instrumented vehicles in the Baltimore area found that 18 percent of total driving time in the area was composed of higher speeds and sharper accelerations than those represented on the FTP test. Also, a 1995 National Research Council report noted that aggressive driving with many accelerations resulted in hydrocarbon and carbon monoxide emissions being 14 and 15 times higher, respectively, than the emissions from average driving over the same 7-mile trip.6 According to the 1995 National Research Council report, “Virtually all motor vehicle testing has been based on a limited set of driving test cycles that inadequately represent current urban driving conditions.” However, one model expert told us that it took more than 1 year to evaluate one component of the model and that collecting vehicle emissions data on large data sets is very costly. EPA officials pointed out 6 Expanding Metropolitan Highways: Implications for Air Quality and Energy Use, Special Report 245, Transportation Research Board, National Research Council, National Academy of Sciences, 1995. Page 18 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a that there is not a consistent definition of what constitutes aggressive driving, that aggressive driving happens only over a portion of the trip and is highly variable among drivers, and that the above observations are not representative of average driving patterns. The Congress has also recognized that the FTP may not reflect actual driving conditions. Concerned about the gap between emissions as measured by the FTP and actual, real-world emissions, in 1990 the Congress added Section 206(h) to the Clean Air Act, which required EPA to review and revise the FTP within 18 months “to insure that vehicles are tested under circumstances which reflect the actual current driving conditions under which motor vehicles are used.” EPA’s October 1996 final rule on FTP revisions7 addressed four emissions-producing activities that, according to the rule’s preamble, are not adequately represented in the current FTP. These emissions-producing activities include (1) aggressive driving behavior (such as high acceleration rates and high speeds), (2) rapid speed fluctuations (such as quick deceleration8), (3) emissions immediately after engine start-up, a period when—because of the fact that engines are designed to operate at higher temperatures—emissions typically bypass emissions controls for an estimated 3 to 5 minutes until the engine reaches normal operating temperatures, and (4) actual air conditioner usage. EPA has not yet revised the MOBILE model to reflect the results of recent studies that have led to these FTP rule revisions but has work ongoing in all four areas. According to agency officials, although adjustments had been made to the MOBILE model for most of these activities prior to issuing the revised FTP rule, given the state of knowledge today, it appears that these activities may not be adequately represented in MOBILE5a. EPA officials told us that incorporating new estimates for these emissions-producing activities would be a high priority for MOBILE6, due to be issued in late 1998. For example, in March 1997 agency officials announced their plans to substantially revise the cold-start segment of the next MOBILE model, moving—for the first time ever—from an areawide, trip-based model to a roadway-specific model that also separately accounts for start-up emissions. Under this revised model, the magnitude of start-up emissions will not depend on vehicle speed or the driving cycle. Instead, EPA is proposing to allow model users to model the emissions impacts of cold 7 Final Regulations for Revisions to the Federal Test Procedure for Emissions From Motor Vehicles, 61 Fed. Reg. 54852 (Oct. 22, 1996). 8 Rapid deceleration causes excess emissions because more fuel is fed into the engine than the engine now demands, resulting in improperly combusted fuel. Page 19 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a starts on the basis of local areas’ estimates of the number of such starts. Additionally, model users will be able to estimate emissions for three different types of roadways—freeways, arterials, and local roadways. Eight Limitations Related In addition to concerns about the representativeness of the FTP to Other Areas of the parameters, the following issues were also identified by model experts, MOBILE Model workgroup participants, and/or stakeholders we contacted. In each instance, EPA officials agreed that the limitation is an area of concern and in most cases noted that the agency has ongoing work to address the issue, which is discussed below. One concern is the representation of high emitters9 in EPA’s MOBILE model database, since the data now indicate that this group of vehicles accounts for a disproportionate amount of an area’s overall emissions and that if this subset of the overall vehicle population is underrepresented, the impact on the emissions estimates can be substantial. For example, Sierra Research testified in 1995 that the worst polluting 22 percent of the vehicles produce about 50 percent of the emissions, and EPA estimates that, overall, from 10 to 30 percent of the vehicles cause the bulk of the pollution problems. As noted by the nonfederal co-chair of the FACA modeling workgroup in a 1993 study, “The general problem of failing to control for significant factors is compounded by the likelihood that a small fraction of the vehicle fleet are currently responsible for a large percentage of vehicle emissions.”10 He told us that he still believes this to be one of the most significant issues facing EPA today, primarily because a very small number of vehicles can potentially be responsible for unusually high levels of pollution. This was also a significant issue for one of the Session Chairs for the Coordinating Research Council’s (CRC) April 1997 Workshop,11 who noted that, in his opinion, this is the single greatest issue that EPA faces—how to identify and repair high emitters and properly represent them in the modeling database. The nonfederal co-chair also noted that if the occurrence of such vehicles is not properly represented in the model, 9 High emitters are generally defined as cars with emissions more than twice the applicable standard and are often described on the basis of the amount of excess emissions they produce. For CO and HC, EPA classifies such excess emitters as high, very high, and super emitters; for NOx, vehicles are either normal or high. 10 R. Guensler, “Data Needs For Evolving Motor Vehicle Emission Modeling Approaches” (Institute of Transportation Studies, Univ. of California - Davis, Aug. 1993). 11 Proceedings of the Seventh Coordinating Research Council’s On-Road Vehicle Emissions Workshop, April 9-11, 1997; published June 30, 1997. Page 20 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a the model’s emissions estimates can be seriously flawed. He and others have concerns that the existing database may underrepresent high emitters because, among other reasons, the owners of such vehicles may avoid surrendering such vehicles for inspection and maintenance (I&M) and other testing at a higher rate than the normal population. As noted in a February 1996 study, “individuals with intentionally tampered or poorly maintained vehicles may be less likely to offer their vehicles for testing.”12 Additionally, a 1993 CARB study of 186 vehicles indicated that high emitters could represent 16.8 percent of the California fleet, or nearly 5 times the assumption in the California model.13 EPA officials have some concerns with the study, its reliance on remote sensing devices, and its applicability to other states. Additionally, EPA officials believe that the larger data sets provided by their ongoing I&M lane testing in three other states properly identify most high emitters. However, they agreed that appropriate representation of high emitters is important to the model’s emissions estimates and noted that this is also a high priority issue currently being addressed by EPA and one of the subgroups of the mobile modeling workgroup. A second concern is the current correction factors for lower volatility fuels and for oxygenated fuels. For example, a February 1997 report by Sierra Research14 found, among other things, that MOBILE5a likely underestimates the impact of low reid vapor pressure15 (RVP) fuels on hydrocarbon and carbon monoxide emissions at temperatures above 75 degrees. The report notes that the correction factor for low RVP fuels has not changed since February 1989, when limited data on fuels with RVPs lower than 9.0 pounds per square inch (psi) caused EPA to place a constraint code in the model precluding users from being able to calculate reductions below this level. The limited data collected since that time indicate that reducing fuel RVP from 9.0 psi to 7.0 psi may reduce hydrocarbon and carbon monoxide exhaust emissions from 18 to 27 percent more than the model estimates, respectively. The January 1997 Auto/Oil Air Quality Improvement Research Program (AQIRP) Final Report16 suggested that reducing fuel volatility by 1 psi, from 9.0 to 8.0 psi, would 12 “Analysis of Causes of Failure in High Emitting Cars,” Publication Number 4637, Health and Environmental Sciences Department, American Petroleum Institute, February 1996. 13 M. Carlock, “An Analysis of High Emitting Vehicles in the On-Road Vehicle Fleet,” California Air Resources Board, 1993. 14 This study was sponsored by the American Petroleum Institute (API). 15 RVP is a measure of fuel volatility, expressed as pounds per square inch, with higher pressures resulting in higher volatility and, therefore, more hydrocarbon emissions. 16 Auto/Oil Air Quality Improvement Research Program Final Report, January 1997. Page 21 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a reduce exhaust CO by 9 percent, exhaust HC by 4 percent, and total evaporative HC by 34 percent (NOx remained unchanged). The RVPs for most fuels used to be higher than 9.0 psi,17 but today they can go lower than 7.0 psi. Similarly, the model currently has no emissions reduction credits for low sulfur fuels (except, according to agency officials, for the lower sulfur effect in reformulated gasoline), although recent studies suggest that lowering the concentration of sulfur in fuel reduces the emissions of hydrocarbons and nitrogen oxides. EPA officials said that they plan to eliminate the constraint code in the next model, MOBILE6, which will allow users to receive credit for correction factors for fuels lower than 9.0 psi; however, they noted that work in this area is still ongoing and that the data on the emissions benefits of lower RVP fuels, as well as low sulfur fuels, are limited. In addition, a 1996 National Research Council study suggested that the model may overestimate the benefits of oxygenated fuels. For example, the study noted that EPA’s MOBILE model “apparently overpredicts the oxygenated fuel effect by at least a factor of two” when the model’s estimate of carbon monoxide reductions is compared with observed data. Similarly, a 1997 study18 of wintertime oxygenated fuels suggested that the observed oxygenated fuel benefits were much lower than the 20 to 30 percent estimated by EPA’s model. EPA officials agreed that this is also an area that needs more study, but one which they plan to address in MOBILE6. A third concern is MOBILE5a’s estimates of emissions system deterioration for vehicles with more than 50,000 odometer miles. This concern stems from studies that have questioned the rate and quantity of the data supporting EPA’s significantly higher rate of emissions system deterioration once vehicles reach 50,000 odometer miles.19 For example, prior to MOBILE5, the model assumed that a vehicle with 100,000 miles emitted about 1.0 grams of hydrocarbons for each mile driven, or about 4 times the amount a new car would emit. However, EPA adjusted the deterioration rates for vehicles with more than 50,000 miles beginning with MOBILE5 (Dec. 1992) so that the MOBILE model’s deterioration formula now calculates that the same car was emitting about 2.0 grams of hydrocarbons for each mile driven, or about 8 times the amount a new car would emit. 17 According to a Senior Environmental Scientist, Health and Environmental Sciences Department, American Petroleum Institute, from 1990 to 1992, average gasoline RVP was about 10 psi. 18 L. Anderson, R. Jones, P. Wolfe, “Assessing the Effectiveness of Oxygenated Fuels Use for the Reduction of Ambient Carbon Monoxide” (Univ. of Colorado at Denver), Seventh CRC On-Road Vehicle Emissions Workshop, April 9-11, 1997. 19 According to EPA, 50,000 miles was the regulatory “useful-life” standard for light-duty vehicles at the time MOBILE5a was developed. Page 22 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a EPA acknowledges that these adjustments were made on the basis of limited data and that only recently have 1990-technology vehicles become old enough to accurately assess their emissions deterioration. An October 1996 Sierra Research study20 of 75 vehicles with over 100,000 odometer miles questioned whether EPA had perhaps adjusted the formula too much, resulting in a model that currently overestimates the emissions from vehicles with 50,000 or more odometer miles. Among other things, the study found that EPA’s current model estimated that 80 percent or more21 of these higher mileage vehicles (which, on average, had accumulated 123,900 odometer miles) would be high emitters, whereas the study found that only 32 percent of the vehicles fell into this category. Similarly, an April 1997 study22 of 227 vehicles (model years 1991 to 1993) with more than 50,000 odometer miles found no significant changes in emissions or deterioration as indicated by the current model. OMS officials said they used all the data that were available to them at the time (1991-1992) to estimate the deterioration rate of such vehicles and that researchers since then had had more time and more vehicles to test than were available to EPA. They also said the model’s correction factor for vehicles with 50,000 or more odometer miles would likely be lowered in MOBILE6, but they were uncertain at the time of our audit how much this emissions estimate would be reduced. A special subgroup of the FACA mobile model workgroup has been established to address this issue, and their work is still ongoing. A fourth concern is MOBILE5a’s emissions credits and assumptions about inspection and maintenance programs. According to a 1995 National Research Council report, vehicle condition—whether the vehicle is well maintained, or has been tampered with or is malfunctioning—is more important than vehicle age in determining emissions. Among other issues, there is a need to update the basic data supporting I&M emissions reduction credits to reflect a growing population of vehicles in which rates of tampering may be diminishing since tampering with newer vehicles 20 This study was prepared by Sierra Research for API using data collected under another API-sponsored study, the High Mileage Vehicle project. 21 This range results from differences in the model’s assumptions about emissions because of the type of fuel-delivery systems used. The model estimates that 80 percent of port fuel-injected vehicles would be high emitters, and that 97 percent of throttle-body fuel-injected vehicles would be high emitters. Carbureted vehicles were not a part of this study because this fuel delivery technology is no longer used on light duty vehicles, according to the study. 22 D. Berens (Ford Motor Company), H. Haskew (GM Powertrain Group), R. Ortega (Chrysler Corporation), “FTP Emissions From 1991-1993MY In-Use Vehicles,” Seventh CRC On-Road Vehicle Emissions Workshop, April 9-11, 1997. Page 23 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a adversely affects gas mileage and vehicle performance. Also, vehicle owners often replace older, carbureted vehicles with newer fuel-injected vehicles. Additionally, according to agency officials, the current model provides no additional I&M credits for vehicles equipped with on-board diagnostics23 (OBD), a requirement for all 1994 and later light duty vehicles and trucks. This vehicle computer technology alerts a car owner when an emissions system malfunctions, permitting quicker repairs than when such malfunctions are identified through an I&M testing program, and diagnostic trouble codes assist mechanics in making better repairs. Additionally, newer vehicles have up to 8 years or 80,000-mile emissions control system warranties for two components (the on-board computer and catalytic converter), which should equate to less-polluting vehicles as a result of more durable emissions control systems and the requirement that manufacturers cover the costs of certain repairs. The current model does not provide specific credits for this growing population of OBD-equipped vehicles designed and believed to have less in-use deterioration than their predecessors. Also, more recent and more complete data are needed on the effectiveness of repairs in an I&M program, including the adequacy and durability of these repairs, actual participation rates, and impact of remote sensing efforts. Except for remote sensing, the current model’s estimates for these parameters is based on aging and limited data.24 For example, EPA has not performed any tampering surveys since 1992, and agency officials said that as a result of this lack of data, the tampering assumptions for MOBILE6 will remain unchanged from MOBILE5a. However, EPA’s goal for MOBILE6 is to provide users with greater flexibility in designing I&M programs, as long as the state or local programs’ estimated I&M credits can be substantiated with state or local data. With respect to newer vehicles equipped with on-board diagnostics, because of the limited data on the longer-term emissions impact of this technology, the agency has provided credit for these OBD-equipped vehicles equal to that provided for operating an enhanced I&M program. EPA officials said that, in addition to the states’ own studies, the agency currently has I&M effectiveness studies being carried out in three states, but they were unsure whether sufficient data would be available in time to further revise the I&M assumptions in MOBILE6. 23 All 1994 and later light duty vehicles and light duty trucks must be equipped with an emissions control diagnostic system capable of identifying deterioration and/or malfunction of the emissions control system, referred to in EPA’s regulations as on-board diagnostics . 24 EPA’s remote sensing data are limited, but recent, according to agency officials. Page 24 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a A fifth concern is the proper representation of diurnal emissions. Diurnal emissions refer only to hydrocarbons and are a form of evaporative emissions that occur when a vehicle is parked and the ambient temperature is fluctuating. For all previous versions of the MOBILE model, the data supporting these 8- to 24-hour emissions estimates were collected during a 1-hour period during which temperatures were forcibly increased over a range of temperatures. More recent testing over 24-hour and longer periods without constraining temperature increases to a one-hour period indicates some differences from the MOBILE5a estimates for such evaporative emissions. Also, evaporative emissions from vehicles with fuel leaks are now believed to be so significant that, for MOBILE6, EPA plans to model these emissions separately from other evaporative emissions. According to EPA’s most recent data, indications are that some vehicles with fuel leaks—similar to super emitters of exhaust/tailpipe emissions—can exceed the evaporative emissions of corresponding vehicles by one to two orders of magnitude. A 1996 automotive industry study25 of 150 vehicles found that 24-hour diurnal emissions ranged from 0.6 grams of HC to 777.2 grams of HC, with vehicles with liquid fuel leaks providing the vast majority of the emissions. EPA officials explained that while they may develop a separate category for some vehicles with significant fuel leaks, this does not necessarily mean there will be a significant overall increase in evaporative emissions estimates because, until more data are collected, there is no clear indication that these emissions were significantly underestimated in prior diurnal estimates. EPA has testing under way to determine how to better define this category of vehicles with significant fuel leaks and also plans tests to estimate their distribution within the current fleet, rate of occurrence as a function of accumulated mileage, vehicle age, and/or vehicle technology, such as fuel tank design. Agency officials are uncertain at this time whether the correction factors for other evaporative emissions estimates will be revised for MOBILE6. A sixth concern is the adequacy of the data supporting MOBILE5a’s assumptions about emissions for in-use26 I&M heavy duty27 vehicles. According to EPA’s June 1994 workshop on state needs, the in-use credits for heavy duty gasoline-powered vehicles are based on data approximately 25 H. Haskew (GM Powertrain Group), “Diurnal Emissions From In-Use Vehicles,” Seventh CRC On-Road Vehicle Emissions Workshop, April 9-11, 1997. 26 In-use emissions estimates attempt to assess actual vehicle emissions over time, mileage, vehicle type, and other factors, such as whether the vehicle is subjected to an I&M program, according to EPA officials. 27 Generally defined as vehicles 8,501 pounds or more in gross vehicle weight. Page 25 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a 20 years old, and there has been much change in the technology and emissions rates of these vehicles since that time. Still, the certification standards are higher for heavy duty vehicles than their light duty counterparts, and they are generally older and driven more miles annually than their light duty counterparts. EPA officials said that testing heavy duty vehicles is difficult and quite expensive and agreed that there is a lack of recent data on the in-use emissions from this category of vehicles once they have been put in service. While some studies are under way, EPA does not envision at this time that significant changes in the in-use emissions rates for heavy duty vehicles will be included in MOBILE6. A seventh concern is the fleet characterization data in EPA’s database, stemming from a concern that much of the data used for MOBILE5a are quite old. For example, MOBILE5a’s estimates are based on the assumption that, on average, light duty vehicles are driven about 14,000 miles annually when new, decreasing to less than 10,000 miles annually after 10 years. More recent data from the U.S. Department of Transportation indicates that passenger cars are driven about 2,000 miles more annually than currently estimated by EPA’s MOBILE model, or nearly a 10-percent increase over MOBILE5a. According to a 1996 report,28 because of the linkage between odometer mileage and I&M program assumptions, a small change in mileage accumulation rates can result in a large impact on emissions estimations. EPA officials pointed out that the agency’s guidance encourages model users to provide their own accumulated mileage estimates; thus, they said the default values for accumulated mileage in MOBILE5a would be a problem only in those cases in which model users fail to provide their own accumulated mileage estimates. Additionally, heavy duty gasoline-powered vehicles, which have higher certification standards than their light duty counterparts, are believed to comprise a significantly larger percentage of the overall vehicle fleet than currently estimated by the MOBILE model. According to OMS, the agency plans to update the fleet characterization data for MOBILE6, including reflecting the increases in the heavy-duty vehicle population. Similarly, another fleet characterization issue involves urban buses. For example, the current model does not have a separate classification for urban buses, although this is a growing vehicle category in many urban areas with unique operating characteristics, such as very frequent starts and stops. EPA officials explained that while buses have not been a separate category in MOBILE5a, EPA plans to expand the current list of 28 J. Heiken, B. Austin, and A. Pollack (Environ International Corporation) and D. Coe, L. Chinkin, and D. Eisinger (Sonoma Technology, Inc.), “Estimation of Local Fleet Characteristics and Activity Data for Improved Emission Inventory Development,” March 18, 1996. Page 26 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a vehicle categories from 8 to 20, one of which will be a separate category for buses. An eighth concern is the level of distinctions in roadway classifications. The MOBILE model was originally designed only for estimating areawide emissions on the basis of assumptions associated with an entire trip. It was not designed for making decisions for various roadway classifications, such as transportation improvement projects for urban interstate, rural arterial, or urban feeder/collector streets. Several model experts have pointed out that the same average travel speed—35 mph, for instance—would indicate smooth traffic flow on a local street but severe congestion on a freeway. EPA officials pointed out that MOBILE6 will allow users to separate start emissions from any linkage to the FTP driving cycle assumptions and will also provide different correction factors for speed and driving cycle for three different types of roadways—freeways, arterials, and local roadways. Additionally, while not planned for MOBILE6, the agency plans to partially fund ongoing research with the Department of Transportation to develop a modal emissions model that may one day allow users to model additional parameters, such as the relative emissions impact of sequencing traffic signals to enhance traffic flow. As noted above, an EPA-sponsored FACA mobile model workgroup made up of representatives from other federal, state, and local government agencies, academia, the automobile and oil industries, environmental groups, and others has been assisting EPA in improving the current model, and much of the research to fill data gaps and update aging databases was still ongoing at the time of our audit. Agency officials said that it is their plan for each new version of the MOBILE model to reflect the most recent testing, data collection, and research that are available. Except for the impact of road grade on emissions and revising the in-use credits for heavy duty vehicles, EPA officials said they plan to address each of the above limitations in the next revision, MOBILE6. However, as discussed below, the agency will not be able to quantify the uncertainty associated with its MOBILE model estimates, primarily because of the complexity and timing of factors affecting vehicle emissions and the high cost of vehicle studies. The Uncertainty Limitation Another significant limitation involves the lack of information about the range, or magnitude, of the uncertainty29 associated with the model’s estimates. Uncertainty occurs for several reasons but often is the result of 29 Traditionally, this has been referred to as the 95-percent confidence interval, indicating with 95-percent certainty that the true number is within a specified range. Page 27 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a omissions, or gaps, in the understanding of emissions-producing activities and of errors in assumptions that are the result of data gaps or data limitations, as well as mathematical and statistical variabilities. However, according to the nonfederal co-chair of the FACA workgroup, information about the uncertainty of the model’s estimates is critical to the ability of policymakers to make good decisions today and for researchers to address the most significant areas in route to a better future model. The co-chair and other model experts told us that the current MOBILE model has no information about, nor estimates of, the uncertainty associated with its predictions. According to one researcher, the model “just produces a number for the user, with no documentation of the reliability of the output.” Similarly, according to a 1994 National Research Council special report30 addressing the MOBILE model, “Uncertainty is pervasive in all three emission modeling components: vehicle activity, activity-specific emission rates, and emission rate correction factors. Uncertainty is compounded in the methodologies used to develop the emission inventory. That is, vehicle activity uncertainty is combined with emission rate uncertainty that has already been combined with correction factor uncertainty.”31 Additionally, the limited work in this area indicates there are significant uncertainties associated with the current MOBILE model’s estimates. For example, one study32 found that “the range of uncertainty is huge” for a change in one variable—average vehicle speed—of the many variables contained in the MOBILE model. According to the study, most model users generally believe that increasing average vehicle speed from 30 mph to 50 mph will reduce vehicle emissions (because of less congested driving, with more driving at cruising speeds). The study noted that EPA’s MOBILE model estimates a 24-percent reduction in carbon monoxide emissions by increasing average vehicle speed from 30 mph to 50 mph. However, when a 95-percent confidence interval is applied, the change in emissions can range from a 72-percent decrease in carbon monoxide emissions to a 30 R. Guensler and D. Sperling, Congestion Pricing and Motor Vehicle Emissions: An Initial Review, Institute of Transportation Studies, University of California at Davis, commissioned paper for the Transportation Research Board, National Research Council, National Academy of Sciences. 31 With respect to MOBILE, according to EPA, only two of these modeling components—emissions rates and correction factors—are subject to concerns about uncertainty; model users supply their own vehicle activity data from other sources. 32 R. Guensler, Vehicle Emission Rates and Average Vehicle Operating Speeds, Ph. D. dissertation, Institute of Transportation Studies, University of California at Davis, 1993, excerpted for the 1994 commissioned paper for the Transportation Research Board, National Research Council, National Academy of Sciences. Page 28 GAO/RCED-97-210 Air Pollution Appendix I Major Limitations in MOBILE5a 75-percent increase in such emissions. Similarly, a 1996 study33 of EPA’s speed correction factors for vehicle exhaust emissions found substantial uncertainty in EPA’s current MOBILE model. Among other things, the study concluded that the MOBILE model may significantly underestimate carbon monoxide and hydrocarbon emissions—“by up to 3 orders of magnitude”—as the model relates to changes in vehicle speed. According to Office of Mobile Sources officials, EPA has been unable to quantify the model’s uncertainty primarily because of the cost and time associated with such quantification, the fact that the on-road vehicle population is a constantly changing universe of differing emissions control devices and levels, technological limitations in measurement devices, and because there is substantial naturally occurring variability in vehicle emissions (leading to further data gaps/limitations). Agency officials pointed out that there is substantial variability across (1) vehicle types (such as model year, emissions control system, engine type), (2) vehicle operating conditions (cold start, load, speed), (3) the external environment (road grade, temperature, humidity, altitude), (4) vehicle fuels (reformulated, oxygenated, reid vapor pressure), and (5) driver behavior (quick starts and stops, timing and frequency of trips). For these reasons, OMS officials told us they do not plan to develop uncertainty ranges with the next revision to the MOBILE model. Similar to the global positioning issue for road grade, they said significant technological advancement may be needed before it becomes cost-effective to address this issue. For example, future vehicles may have on-board computers with the ability to instantaneously record and later report emissions under different operating scenarios. EPA officials pointed out that it is their plan, at some point in the future, to report uncertainty ranges for some model estimates. They said they are currently saving both qualitative and quantitative descriptors for the data being collected by and for the agency in order to perform these calculations in the future. 33 M. Kini, Probabilistic Modeling of Exhaust Emissions From Light Duty Gasoline Vehicles, Graduate Thesis, Department of Civil Engineering, North Carolina State University, Oct. 1996. Page 29 GAO/RCED-97-210 Air Pollution Appendix II Comments From the Environmental Protection Agency Page 30 GAO/RCED-97-210 Air Pollution Appendix II Comments From the Environmental Protection Agency Page 31 GAO/RCED-97-210 Air Pollution Appendix III Objectives, Scope, and Methodology The Chairman, Subcommittee on Oversight and Investigations, House Committee on Commerce, asked us to (1) describe the major limitations in the current version of EPA’s MOBILE model and (2) describe EPA’s process for improving both current and future versions of the MOBILE model. To describe the major limitations in the current model, we obtained and reviewed the MOBILE5a User’s Guide; EPA/OMS model documentation; the most recent published sensitivity analyses (MOBILE4.1, May 1990); relevant EPA guidance and memorandums on the MOBILE model; selected vehicle studies; the results of stakeholders meetings about the model; and the charter, objectives, minutes, and proceedings of the EPA-Federal Advisory Committee Act mobile modeling workgroup. We also reviewed five electronic databases1 for studies pertaining to the EPA MOBILE model; attended one mobile sources symposium where modeling issues were discussed; and attended the March 1997 FACA public workshop. We also obtained and discussed studies relating to potential MOBILE model limitations with selected representatives of state and local agencies, academia, industry, environmental groups, consulting firms, and other government agencies. Additionally, we interviewed officials and obtained documents from EPA’s Office of Mobile Sources in Ann Arbor, Michigan; Office of Research and Development in Research Triangle Park, North Carolina, and Athens, Georgia; and the EPA Science Advisory Board in Washington, D.C. We also discussed model limitations with individuals identified to us by EPA or other representatives, as well as through our own efforts as noted above. To describe EPA’s process for improving both current and future versions of the MOBILE model, we obtained and discussed information from knowledgeable EPA/OMS air quality officials relative to ongoing activities and documented plans for making model revisions. We also discussed EPA’s past, ongoing, and planned actions with representatives of academia, industry, environmental groups, consulting firms, and other government agencies and observed one process—EPA’s open solicitation of input by 1 The electronic databases reviewed included “Enviroline,” which covers more than 5,000 international primary and secondary environmental publications on all aspects of the environment; “EiCompendex Plus,” the electronic version of The Engineering Index, which provides worldwide coverage of approximately 2,600 journals and selected government reports and books on the environment and other issues; “Pollution Abstracts,” a leading resource for references to environment-related literature on pollution, its sources, and its control; “Energy Science and Technology,” one of the world’s largest sources of literature references on energy and related topics, including the environment, with coverage of journal articles, report literature, conference papers, books, patents, dissertations, and translations; and EPA’s Technology Transfer Network, a worldwide network of electronic bulletin boards providing information and technology exchange in areas pertaining to air pollution control, with emphasis on EPA’s Mobile Sources Information bulletin board, which covers information pertaining to mobile source emissions, regulations, test results, models, and guidance. Page 32 GAO/RCED-97-210 Air Pollution Appendix III Objectives, Scope, and Methodology external stakeholders not involved in the model’s development—at work. We also obtained documents and discussed EPA’s process for improving models in general with the Science Advisory Board and EPA’s Office of Research and Development. We conducted our review from October 1996 though August 1997 in accordance with generally accepted government auditing standards. Page 33 GAO/RCED-97-210 Air Pollution Appendix IV Major Contributors to This Report Lawrence J. Dyckman, Associate Director Resources, William F. McGee, Assistant Director Community, and Judy K. Pagano, Senior Operations Research Analyst Economic James R. Beusse, Evaluator-in-Charge Hamilton C. Greene, Jr., Staff Evaluator Development DeAndrea M. Leach, Staff Evaluator Division, Washington, D.C. (160369) Page 34 GAO/RCED-97-210 Air Pollution Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Air Pollution: Limitations of EPA's Motor Vehicle Emissions Model and Plans to Address Them
Published by the Government Accountability Office on 1997-09-15.
Below is a raw (and likely hideous) rendition of the original report. (PDF)