United States General Accounting Office GAO Report to the Chairmen, Committee on Commerce, Science, and Transportation, U.S. Senate, and the Committee on Commerce, House of Representatives September 1997 CONSUMER PRODUCT SAFETY COMMISSION Better Data Needed to Help Identify and Analyze Potential Hazards GAO/HEHS-97-147 United States GAO General Accounting Office Washington, D.C. 20548 Health, Education, and Human Services Division B-274042 September , 1997 The Honorable John McCain Chairman, Committee on Commerce, Science, and Transportation United States Senate The Honorable Thomas J. Bliley, Jr. Chairman, Committee on Commerce House of Representatives Every year, children are hurt while using playground equipment; users or spectators are injured when fireworks explode unexpectedly; and homeowners are cut while operating chain saws and lawn mowers. Created to protect consumers from “unreasonable risk of injury,” the U.S. Consumer Product Safety Commission (CPSC) oversees these and about 15,000 other consumer products ranging from kitchen appliances and children’s toys to hot tubs and garage door openers. With a budget of about $42.5 million, CPSC carries out its mission by (1) enforcing federal consumer product regulations (by recalling products from store shelves, for example) and (2) conducting projects to address products with potential hazards not covered by existing regulations.1 These projects may result in CPSC issuing new regulations concerning specific products, assisting in the development of voluntary industry standards, or providing information to consumers about how to use the products safely. Contending that the agency is ineffectively allocating its resources, CPSC’s critics have voiced dissatisfaction with the selection of certain agency projects and have questioned the validity of CPSC’s risk assessment and cost-benefit analyses supporting those projects. In addition, congressional and interest group critics have questioned the agency’s procedures for ensuring the accuracy of manufacturer-specific information before releasing it to the public, contending that such releases can mar the reputation of responsible corporate citizens. In light of these concerns, you asked us to review CPSC’s project selection, use of cost-benefit analysis and risk assessment, and information release procedures. Specifically, this report (1) identifies the criteria CPSC uses to 1 Projects vary widely in scope, and CPSC has no standard definition of what constitutes a project. For our review, we defined a “project” as work CPSC conducted on any specific consumer product that was associated with a potential hazard or hazards not covered by existing regulation. Page 1 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 select projects and reviews the information it relies upon in making these choices, (2) assesses the information CPSC draws on to perform risk assessment and cost-benefit analyses and evaluates the agency’s methodology for conducting cost-benefit analyses, and (3) describes CPSC’s procedures for releasing manufacturer-specific information to the public and reports whether evidence exists that CPSC violated its statutory requirements concerning the release of such information. To address these objectives, we reviewed internal CPSC documents, relevant legislation and regulations, and the literature on cost-benefit analysis and on consumer product safety issues. We interviewed CPSC commissioners and staff, four former commissioners, consumer advocates, industry representatives, and outside experts to obtain their perspectives on CPSC’s work. We identified CPSC projects by compiling from various agency documents a list of 115 potential product hazards examined by the agency from January 1990 to September 30, 1996, and we reviewed available agency documentation on each of these projects. We examined the agency’s internal databases to obtain project information and to assess the agency’s information on product hazards. In reviewing CPSC’s cost-benefit analyses, we consulted with experts to develop objective criteria that reflected elements commonly used in the evaluation of cost-benefit analyses. These evaluation questions were designed to elicit whether CPSC conducted comprehensive analyses and reported them in sufficient detail. However, they do not make up a complete measure of the quality of a cost-benefit analysis; for example, our evaluation assessed whether these elements were included but not how well they were measured or incorporated. We reviewed CPSC’s internal procedures concerning information clearance and release, and we examined relevant legal cases in this area. For more detailed information on our scope and methodology, see appendix I. Although CPSC has established criteria to help select new projects, with the Results in Brief agency’s current data these criteria can be measured only imprecisely, if at all. The criteria for selecting projects include the number of product-related injuries, chronic illnesses, and deaths. However, although CPSC has described itself as “data driven,” its information on product-related injuries and deaths is often sketchy. For example, because the agency’s measure of injuries generally includes only hospital emergency room reports, CPSC has an incomplete picture of injuries. Similarly, CPSC’s data on product-related deaths understate the total number of deaths and are available to the agency only with a 2-year time Page 2 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 lag. In addition, CPSC does not maintain a comprehensive list or description of either its past or ongoing projects. This makes it more difficult not only for agency management to monitor current projects but also for staff and commissioners to assess and prioritize the need for new projects in different hazard areas. As a result, CPSC has insufficient data on both internal agency efforts and external product hazards to assess the impact and cost of each project, either when it is selected or after it has been implemented. To help evaluate alternative methods of addressing potential hazards, CPSC may perform a risk assessment to estimate the likelihood of injury associated with a hazard or conduct a cost-benefit analysis to assess the potential effects of a proposed regulation. Although CPSC does not complete either a risk assessment or cost-benefit analysis for every project, the agency conducts these analyses more often than it is required to by law. Nevertheless, CPSC’s data are often insufficient to support a thorough application of these analytical techniques—a problem that frequently arises in doing both risk assessment and cost-benefit analysis. To evaluate relative risks, it is usually necessary to have information on how many consumers use the product—information that CPSC frequently does not have. In addition, risk assessment of consumer products requires measurement of the number of harmful incidents. CPSC’s imprecise and incomplete death and injury data make risk assessment and cost-benefit analysis at best less reliable and at worst impossible to do. Furthermore, the cost-benefit analyses conducted by CPSC between 1990 and 1996 were often not comprehensive, and the reports on these analyses were not sufficiently detailed. For example, experts generally agree that sensitivity analysis—a technique that enables the reader to determine which assumptions, data limitations, or parameters are most important to the conclusions—should be incorporated in cost-benefit analyses. Most of CPSC’s cost-benefit analyses did not include such information. We are making recommendations to the Chairman of CPSC to improve the agency’s project selection and cost-benefit analyses. CPSC has established procedures to implement statutory requirements concerning the release of manufacturer-specific information. When releasing information to the public that identifies a specific manufacturer—for example, in a safety alert or recall notice—CPSC is required to verify the information and allow the manufacturer an opportunity to comment. Evidence from the industry and from legal cases suggests that CPSC has met its statutory requirements in this area. Individuals within CPSC, as well as some industry representatives and Page 3 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 consumer groups, expressed dissatisfaction with the requirements of this law. Some of these individuals have proposed statutory changes that range from reducing to expanding the current requirements. CPSC was created in 1972 under the Consumer Product Safety Act (P.L. Background 92-573) to regulate consumer products that pose an unreasonable risk of injury, to assist consumers in using products safely, and to promote research and investigation into product-related deaths, injuries, and illnesses. CPSC currently has three commissioners, who are responsible for establishing agency policy.2 One of these commissioners is designated the chairman; the chairman directs all the executive and administrative functions of the agency. The Consumer Product Safety Act consolidated federal safety regulatory activity relating to consumer products within CPSC. As a result, in addition to its responsibilities for protecting against product hazards in general, CPSC also administers four laws that authorize various performance standards for specific consumer products. These laws are the Flammable Fabrics Act (June 3, 1953, c.164), which authorizes flammability standards for clothing, upholstery, and other fabrics; the Federal Hazardous Substances Act (P.L. 86-613), which authorizes the regulation of substances that are toxic, corrosive, combustible, or otherwise hazardous; the Poison Prevention Packaging Act of 1970 (P.L. 91-601), which authorizes requirements for child-resistant packaging for certain drugs and other household substances; and the Refrigerator Safety Act of 1956 (Aug. 2, 1956, c.890), which establishes safety standards for household refrigerators. In fiscal year 1997, CPSC carries out this broad mission with a budget of about $42.5 million and a full-time-equivalent staff of 480. As figure 1 shows, after adjusting for inflation, the agency’s budget has decreased by about 60 percent since 1974. Similarly, CPSC’s current staffing level represents 43 percent fewer positions as compared with the agency’s 1974 staff. 2 The Consumer Product Safety Act provides for the appointment of five commissioners by the President of the United States for staggered 7-year terms. However, since 1986, no more than three commissioners have served at one time. Page 4 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Figure 1: CPSC Funding, in Inflation-Adjusted Dollars, 1974-96 CPSC Funding (in Thousands) 90,000 80,000 70,000 60,000 50,000 40,000 30,000 20,000 10,000 0 1974 1976 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 Year Notes: Budget figures were obtained from CPSC. The 1993 budget figure excludes a special appropriation for office space relocation. Budget figures were adjusted for inflation using the Gross Domestic Product deflator for federal nondefense spending, with 1992 as the base year. CPSC uses a number of regulatory and nonregulatory tools to reduce injuries and deaths associated with consumer products. Under several of the acts that it administers, CPSC has the authority to issue regulations that establish performance or labeling standards for consumer products. For example, in 1993, CPSC issued regulations under the Consumer Product Safety Act requiring disposable cigarette lighters to be child-resistant. If CPSC determines that there is no feasible standard that would sufficiently address the danger, CPSC may issue regulations to ban the manufacture and distribution of the product. In addition, under the Consumer Product Safety Act, if a product violates a safety regulation or presents a “substantial hazard,” CPSC may order a product recall, in which the item is removed from store shelves and consumers are alerted to return the item Page 5 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 for repair, replacement, or refund.3 CPSC can also impose civil penalties for violations of federal safety standards. Although CPSC has these broad regulatory powers, much of the agency’s efforts are carried out using nonregulatory methods. In addition to federally mandated product safety standards, many consumer products are covered by voluntary standards. These voluntary standards, which are often established by private standard-setting groups, do not have the force of law. However, many voluntary standards are widely accepted by industry.4 The 1981 amendments to the Consumer Product Safety Act require CPSC to defer to a voluntary standard—rather than issue a mandatory regulation—if CPSC determines that the voluntary standard adequately addresses the hazard and that there is likely to be substantial compliance with the voluntary standard. As a result, voluntary standards development is an important tool in CPSC’s hazard-reduction efforts. For example, in 1996 CPSC helped a private group develop a voluntary standard to address the risk of children getting their heads stuck between the slats of toddler beds, and in 1995 CPSC assisted a standard-setting group in upgrading safety standards to prevent fires associated with Christmas tree lights. CPSC also addresses product hazards by providing information to consumers on safety practices that can help prevent product-related accidents. For example, to encourage consumers to use electricity safely—and particularly to promote the use of ground fault circuit interrupters—CPSC conducted a far-reaching publicity campaign that included radio public service announcements, messages printed on carryout bags for hardware stores, a joint press conference with industry representatives, presentations on television’s Home Shopping Network, and promotional letters to real estate and home inspection associations. In addition to its own active efforts to disseminate information, CPSC provides considerable amounts of information in response to requests from the public. Like other federal agencies, CPSC must comply with the Freedom of Information Act (FOIA) when responding to requests from the public for information. A notable feature of FOIA is its presumption in favor of disclosure: any person has the right to inspect and copy any government 3 In practice, CPSC rarely uses its regulatory power to order a recall, but works cooperatively with manufacturers to carry out recalls. 4 Voluntary standards may benefit manufacturers by giving consumers added confidence in a product, providing some degree of protection from product liability, and allowing manufacturers to benefit from the safety expertise developed by voluntary standards groups. In addition, although federal law does not compel manufacturers to comply with voluntary standards, state or local regulations may incorporate some voluntary standards regarding consumer products, and some retailers prefer to carry only those goods that comply with the applicable voluntary standards. Page 6 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 records unless the documents requested fall within one of the exemptions to the act (for example, disclosure of trade secrets). FOIA requests may come to CPSC from regulated industries, the press, consumer groups, or individuals. During calendar year 1995, CPSC responded to 16,424 formal requests made under FOIA. CPSC’s resource base and extensive jurisdiction require the agency to select among potential product hazards. New initiatives may come to CPSC in several ways. First, any person may file a petition asking CPSC to issue, amend, or revoke a regulation. Petitions, which can be as simple as a letter or as formal and detailed as a legal brief, have come to CPSC from doctors and nurses, consumers and advocacy groups, and industry representatives. CPSC may grant or deny a petition either in full or in part. Even when CPSC denies a petition and declines to issue a regulation, it may still begin a project to address the hazard by promoting a voluntary standard or conducting a consumer education campaign. For example, a project on heat tapes (heated wraps for exposed pipes) originated with a petition from a concerned consumer. CPSC denied the petition for a mandatory standard, but conducted a research study and review of the existing voluntary standard for heat tapes. Second, CPSC receives some product hazard projects from the Congress. The Congress may require CPSC to study a wide-ranging product area. For example, the Consumer Product Safety Improvement Act of 1990 resulted in a large body of work on products affecting indoor air quality, including wood stoves, kerosene heaters, and carpets. The Congress may also direct CPSC to impose a specific regulation, such as when it directed CPSC to require additional labeling on toys intended for children aged 3 to 6 warning parents of possible choking hazards when the toy is used by children under age 3.5 Finally, CPSC commissioners and agency staff may initiate projects or suggest areas to address. CPSC gathers death and injury information to help identify potential product hazards and also obtains input from the public. The agency maintains a toll-free hot line and an Internet site that the public can use to notify agency staff of a possible product hazard. In addition, CPSC holds public meetings to get input on possible hazards to address and on which hazards should receive priority. For example, CPSC increased its efforts to remove drawstrings from children’s clothing after receiving a letter from a woman whose daughter was strangled when her jacket string caught on a playground slide. 5 This mandate was imposed in the Child Safety Protection Act (P.L. 103-267, June 16, 1994). Page 7 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 The selection of projects to address new product hazards takes place at different levels of the agency throughout the year. For a petition, the commissioners decide whether the product hazard warrants further agency involvement, and the commissioners vote on whether to grant or deny the petition. If a project is believed to have a high potential for regulatory action or involve a substantial amount of agency resources, the commissioners decide whether to pursue it. Projects of this caliber are often noted in the agency’s annual budget and operating plan, which must also be approved by commissioner vote.6 Staff request a decision on such a project by preparing a briefing package about the product hazard for the commissioners, who vote to begin the regulatory process, take some other action, or terminate the project. Agency staff generally may decide to initiate projects that are unlikely to result in regulation, and no briefing package is sent to the commissioners for a vote.7 (Of the 115 CPSC projects we identified, 80 (70 percent) were detailed in briefing packages.) The scope of the agency’s projects varies greatly, and CPSC has no standard definition of what constitutes a project. A project might cover general product areas, such as fire hazards, or address only a specific product, like cigarette lighters. A project might require undertaking an extensive research study or providing technical assistance to a group that is developing a voluntary standard. The bulk of CPSC’s workload is made up of projects selected by the agency CPSC Has Limited rather than by the Congress. CPSC has established criteria to help in project Information Available selection, such as the numbers of deaths and injuries associated with a to Assist in Project product. However, CPSC is unable to accurately measure these criteria because its data on potential hazards are incomplete. In addition, CPSC Selection does not maintain systematic information on past and ongoing projects, which makes it difficult to assess and prioritize the need for new projects in different hazard areas. The lack of comprehensive data on individual product hazards and on agency initiatives raises questions about CPSC’s ability to evaluate its own effectiveness—which it is now required to do under the Government Performance and Results Act of 1993 (the Results Act). 6 Commission votes are also taken in several other instances, such as initiating a regulation, the use of the agency logo by outside groups, and the application of civil penalties. 7 For more information about the agency’s organization and structure, see app. II. Page 8 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Agency Has Considerable CPSC has wide latitude over which potential product hazards it targets for Discretion in Project regulatory and nonregulatory action. Although it has little or no discretion Selection over projects mandated by the Congress, CPSC can choose to accept or reject suggestions that are submitted by petition or proposed by the agency staff. As shown in figure 2, 59 percent of CPSC projects were initiated by CPSC, 30 percent originated from a petition, and about 11 percent resulted from congressional mandates. Of the 115 projects the agency worked on from January 1, 1990, to September 30, 1996, 97 (about 90 percent) were chosen by the agency.8 Data were unavailable to assess the extent to which staff suggestions for projects were accepted or rejected. Of the petitions filed with CPSC between January 1, 1990, and September 30, 1996, 60 percent resulted in projects (32 percent by granting the petition in whole or in part, and 27 percent by denying the petition to establish a mandatory regulation but creating a nonregulatory project). In 27 percent of cases, CPSC decided that no action was needed or that existing actions or standards were sufficient to address the issue raised by the petition. In the remaining cases, a decision is still pending or the petition was withdrawn before a decision was rendered. Figure 2: Origination of CPSC Projects, January 1, 1990-September 30, 1996 Petition 30.0% Mandate 11.0% Internal 59.0% CPSC Project Selection CPSC has established criteria for setting agency priorities and selecting Criteria Open to potential hazards to address. These criteria, which are incorporated in Differences in Emphasis agency regulations, include the following: and Interpretation • the frequency of injuries and deaths resulting from the hazard; • the severity of the injuries resulting from the hazard; 8 For a listing of these 115 projects, see app. III. Page 9 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 • addressability—that is, the extent to which the hazard is likely to be reduced through CPSC action—agency regulations note that the cause of the hazard should be analyzed to help determine the extent to which injuries can reasonably be expected to be reduced or eliminated through CPSC action; • the number of chronic illnesses and future injuries predicted to result from the hazard; • preliminary estimates of the costs and benefits to society resulting from CPSC action; • unforeseen nature of the risk—that is, the degree to which consumers are aware of the hazard and its consequences; • vulnerability of the population at risk—whether some individuals (such as children) may be less able to recognize or escape from potential hazards and therefore may require a relatively higher degree of protection; • probability of exposure to the product hazard—that is, how many consumers are exposed to the potential hazard, or how likely a typical consumer is to be exposed to the hazard; and • other—additional criteria to be considered at the discretion of CPSC. Vulnerable Populations, CPSC’s regulations allow for considerable freedom in applying these Numbers of Deaths and criteria; commissioners and staff can base their project selections on what Injuries, and Causality they perceive as the most important factors. For example, the regulations Emphasized Over Other do not specify whether any criterion should be given more weight than the Selection Criteria others, nor must all criteria be applied to every potential project. Indeed, our interviews with present and former commissioners and our review of CPSC briefing packages revealed a pattern in which three criteria—the numbers of deaths and injuries, the causality of injuries, and the vulnerability of the population at risk—were more strongly emphasized than the others. In addition, each of the commissioners we interviewed identified some criteria as being more important than others for project selection. For example, one commissioner indicated that the number of deaths and injuries was most important, while another commissioner included awareness of the hazard in a list of several criteria she believed were most important. However, there was considerable agreement among the commissioners about the importance of several criteria. The commissioners cited two criteria—vulnerability of population and number of deaths and injuries—as especially important for project selection. In addition, several—but not all—commissioners emphasized causality of injuries. None of the other criteria was emphasized by more than one or two commissioners. Page 10 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Because the commissioners use their judgment in applying these criteria, there is no systematic checklist or scoring system that would enable us to determine which factors were considered most important for a particular product. However, information related to some or all of these criteria is sometimes contained in briefing packages and other documents. Our review of CPSC project documentation showed that information on vulnerable populations and the numbers of deaths and injuries associated with the product was likely to be compiled at some time during the project, but information associated with other criteria was less likely to be documented. For example, of the 115 projects we reviewed, death and injury information was available in 97 cases. However, only 26 cases included information on exposure to the hazard, a less-emphasized criterion. Although data were insufficient to compare the universe of possible projects with the ones selected by CPSC, the characteristics of CPSC projects appear generally consistent with the stronger emphasis on death and injury data and on vulnerable populations expressed by the current and former commissioners. For example, while 76 of the 115 projects we examined were directed at least partially at a vulnerable population group, only 13 projects mentioned chronic illness. However, although the number of deaths and injuries associated with product hazards was almost always available in project documentation, there was no pattern of only those projects with high numbers of injuries or deaths being selected. Of the 97 projects that had death and injury statistics, 19 showed fewer than 50 injuries and/or deaths associated with the product. The estimated number of annual injuries associated with product hazards ranged from 1 to 162,100 (for baseball injuries), and the estimated number of deaths associated with product hazards ranged from zero to a high of 3,600 annually (for smoke detectors). This wide range is consistent with CPSC staff’s statement that there is no threshold for the number of deaths and injuries that would require acceptance or rejection of a project. Commissioners Interpret Some Although the commissioners and former commissioners we interviewed Selection Criteria Differently generally agreed on the criteria they emphasized for project selection, they expressed very different views on how some of these criteria should be interpreted. For example, several commissioners viewed vulnerable populations as focusing on children, while others highlighted additional segments of the population that they considered vulnerable. One commissioner also listed low-income and poorly educated consumers as vulnerable populations, and another expressed concern that the elderly were especially vulnerable to injury from product hazards. Project Page 11 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 documentation focused on children more frequently than on other population segments thought to be at special risk. Many projects we examined contained no information in the documentation that indicated a particular population was being considered vulnerable. However, of the 76 projects for which information was available on special populations, 69 (91 percent) mentioned children.9 Industry observers, consumer advocates, current and former commissioners, and others expressed widely diverging views on how to apply causality of injuries in selecting projects. All seven commissioners we interviewed mentioned this as an important criterion, and several stressed causal factors. A major issue surrounding the application of causality is determining the appropriate level of protection the agency should provide when a product hazard results, at least in part, from consumer behavior. For example, a consumer advocate stated that regulatory action may be necessary whatever the cause of the incident if children who were incapable of protecting themselves get hurt. Similarly, another individual told us that CPSC should deal with potential hazards on the basis of the behavior that actually took place, not the behavior that might be expected or considered reasonable. However, other individuals asserted that CPSC should address only those hazards that result from products that are defective—that is, products that create a hazard even when used as intended by the manufacturer. Some industry representatives stated that it was inappropriate for CPSC to take action concerning a product if the product was “misused” by the consumer. Complicating this debate is the difficulty of defining misuse of the product or negligence of the consumer. For example, the appropriate degree of parental supervision is frequently an issue with children’s products. One of the agency’s more controversial projects illustrates this point. CPSC staff conducted a project to investigate the deaths of children using baby bath seats or rings. In these incidents, infants slipped out of the seat and drowned in the bathtub when the parent or caregiver stepped out of the room and left the child unsupervised, despite warning signs on the seats not to leave children unattended. The Commission disagreed on the proper course of action, largely because of differing views on causality. In 1994, the staff recommended that the Commission issue an Advance Notice of Proposed Rulemaking (ANPR), the first step in the regulatory process. The staff argued (and one commissioner agreed) that some parents will leave a 9 Many of these projects involved children’s products, for which only children would be affected, such as cribs or baby walkers. Other projects dealt with products the general public would be exposed to but for which children would be more likely to receive injury, including mouthwashes with high concentrations of alcohol and automatic garage doors that could crush a child when they close. Page 12 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 young child alone in the bathtub regardless of a warning not to. However, in voting against issuing an ANPR, the other two commissioners stated that they believed regulation was not appropriate because the lack of supervision, not the product, caused the tragedies. CPSC staff have also encountered other instances in which the behavior of consumers might be viewed as inappropriate. For example, the role of alcohol and drug use in accidents can also raise questions about the appropriate level of regulatory protection.10 In addition, a 1991 CPSC study found that at least 33 percent of bicycle accidents involved behaviors such as performing stunts and going too fast. Similarly, a 1991 CPSC study of fires associated with heat tapes found that at least 38 percent of the heat tapes had been installed improperly. In each of these cases, no regulatory action was taken; in the case of bicycles, the staff did recommend increasing efforts to encourage consumers to take safety precautions such as using lights at night and wearing helmets. Data Systems Provide CPSC uses data from internal management systems and from external Insufficient Information to sources to assist in project selection. CPSC collects information on Measure Selection Criteria, product-related deaths and injuries to provide information for project selection as well as to perform risk assessments and cost-benefit analyses. Monitor All Projects, or Furthermore, the agency maintains a computerized management Evaluate Results information system (MIS) that contains information on some of its major activities and is used by the agency to develop its annual budget. Both these internal and external data are of limited value. The inadequacy of the information raises questions about CPSC’s ability to make informed project selection decisions so that agency resources are being spent efficiently. Significant Gaps Exist in CPSC has developed a patchwork of independent data systems to provide CPSC’s Data on information on deaths and injuries associated with consumer products. To Product-Related Injuries and obtain estimates of the number of injuries associated with specific Deaths consumer products, CPSC relies on its National Electronic Injury Surveillance System (NEISS). NEISS gathers information from the emergency room records of a nationally representative sample of 101 hospitals. CPSC also obtains information on fatalities by purchasing a selected group of death certificates from the states. It supplements this information with anecdotal reports from individual consumers and with data from private organizations such as fire-prevention groups and poison control centers. 10 Alcohol and drug use contributed to an estimated 1.35 million injury-related visits to hospital emergency departments in 1992. See Cheryl R. Nelson and Barbara J. Stussman, “Alcohol- and Drug-Related Visits to Hospital Emergency Departments: 1992 National Hospital Ambulatory Medical Care Survey,” Advance Data, No. 251 (Aug. 10, 1994). Page 13 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Because neither NEISS nor death certificate data provide detailed information on hazard patterns or injury causes, CPSC investigates selected incidents to obtain more detailed information. In addition, CPSC sometimes uses mathematical modeling techniques or conducts special surveys to obtain information on product exposure. (For more information on CPSC’s data sources, see app. V.) CPSC’s data give the agency only limited assistance in applying its project selection criteria. (These criteria, the measures used for each, and major data limitations are given in table 1.) CPSC’s injury and death data allow the agency to piece together at best an incomplete view of the incidents that result from consumer product hazards. Product-related injuries may be treated in a variety of ways—in an emergency room, in a physician’s office, or through an outpatient clinic, for example. As figure 3 illustrates, CPSC obtains systematic surveillance information only on deaths and on injuries treated in the emergency room; injuries treated in other settings (such as physicians’ offices) are not represented in CPSC’s surveillance data. Page 14 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Figure 3: Types of Death and Injury Data Covered by CPSC’s Systematic Treated in Surveillance Information Physician’s Death Treated in Office ER Treated in Hospitalized: Outpatient Treated at Admitted Clinic Home Through ER Hospitalized: Admitted Not Near Through Treated Miss Physician Partially Covered Covered Not Covered Note: Death data are considered partially included because CPSC obtains death certificates for selected causes of death. For the other sources of treatment, CPSC may obtain some anecdotal information from consumer reports or other sources (see app. V). The actual numbers of injuries treated in each setting is unknown. A “near miss” refers to an incident in which a product-related injury nearly occurred but was narrowly averted. In its regulations that address priority-setting, CPSC states that such incidents can be as important as actual injuries in identifying potential hazards. Page 15 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Table 1: CPSC Regulatory Priority-Setting Criteria and Systematic Criterion Measure used by CPSC Major limitations Information Available Number of deaths Number of reported deaths Incomplete because not all where consumer product certificates are gathered and involvement can be inferred. not all product-related incidents are coded. Number of injuries Estimated number of injuries Generally omits injuries treated involving consumer products in other settings. treated in emergency rooms. Severity of injuries Estimated percentage of Not representative of the emergency-room-treated severity of all injuries treated in injuries requiring all settings. hospitalization. Chronic illnesses Some limited information from Little systematic information. emergency room diagnosis information. Predicted future injuries Prediction based on NEISS Of questionable validity data. because of changes in medical care over time. Vulnerable populations Percentage of Incomplete. Information emergency-room-treated available only on age, not on injuries or deaths involving other vulnerable populations, children or the elderly. such as persons with disabilities. Exposure Various measures, including Exposure surveys are time sales, estimated products in consuming and expensive; not use, and population. done for all projects; done only after project is well under way. Addressability/ Judgment based on cause of Often impossible to make an causation incident, hazard pattern, and informed judgment until project information on product design; is well under way; investigations of selected investigations are time incidents may be conducted to consuming and expensive. obtain this information. Preliminary cost-benefit Results of analysis. Quality data are frequently not analysis available; limited accuracy in early stages of project. CPSC staff identified the lack of data on injuries treated in physicians’ offices and other settings as a key concern. Because CPSC’s data sets reveal only a portion of the injury picture, the agency underestimates the total numbers of deaths and injuries associated with any given consumer product. The extent of this undercount is unknown. For example, researchers report widely varying estimates of the percentage of injuries that are treated in emergency rooms as opposed to other medical settings. For example, a 1991 study by researchers at RAND found that Page 16 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 approximately 65 percent of injuries were treated in the emergency room.11 However, recent data indicate that the number of injury-related visits to physicians’ offices alone were more than double the number of injury-related visits to the emergency room.12 CPSC’s estimates of product-related deaths are also undercounted, for two reasons. First, for budgetary reasons, the agency purchases only a subset of the total number of death certificates from states. Second, CPSC death counts include only those cases in which product involvement can be inferred from the information on the death certificate, and in some cases, product-related information is not recorded. Even if a reliable figure was available to determine the exact percentage of product-related injuries that were treated in emergency rooms, this percentage would not necessarily apply to any specific type of product-related injury. For example, even if it was established that 40 percent of all product-related injuries were treated in emergency rooms, the percentage of bunk bed injuries treated in emergency rooms might be much larger or smaller. The setting in which injuries are treated depends on a wide array of factors that vary among individuals, across geographic regions, and among different types of injuries. Research indicates that African Americans are more likely to use the emergency room than Caucasians are.13 Access to the emergency room or to a physician also depends on the type of medical insurance a person has. For example, health maintenance organizations (HMO) often place restrictions on reimbursement for emergency room care, and HMO membership as a percentage of the total population varies widely from state to state.14 In addition, injuries that occur at night, when most physicians’ offices are closed, may be more likely to be treated in the emergency room. As a result, it is unlikely that CPSC could approximate the number of injuries associated with a specific product by using data that apply to all consumer products as a group. The incompleteness of CPSC’s injury information also hampers its ability to reliably discern long-term trends in injuries, which is not only a criterion 11 Deborah R. Hensler, and others, Compensation for Accidental Injuries in the United States (Santa Monica, Calif.: RAND, 1991). 12 See Susan M. Schappert, “National Ambulatory Medical Care Survey: 1992 Summary,”Advance Data No. 253 (Aug. 18, 1994), and Linda F. McCaig, “National Hospital Ambulatory Medical Care Survey: 1992 Emergency Department Summary,” Advance Data, No. 245 (Mar. 2, 1994). 13 See Linda F. McCaig, “National Hospital Ambulatory Medical Care Survey: 1992 Emergency Department Summary,” p. 1. 14 1996 figures from InterStudy show that while only 1.2 percent of individuals in Mississippi belonged to HMOs, 44.8 percent of individuals in Oregon were HMO members. See The InterStudy Competitive Edge: HMO Industry Report 6.2, InterStudy Publications (1996), p. 28. Page 17 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 for project selection but also an important factor for evaluating the success of CPSC’s injury-reduction efforts and determining the need for follow-up actions. The relative sizes of the pieces of the injury puzzle in figure 3 are unknown but appear to change over time. For example, hospitalizations decreased by 5 percent on a per capita basis between 1982 and 1994, while between 1983 and 1993, hospital outpatient clinics saw a 53-percent increase in visits on a per capita basis.15 As a result, it is impossible to determine whether any change in the number of emergency room visits represents a true change in injuries or a shift to other medical settings.16 According to CPSC staff, identifying chronic illnesses associated with consumer products is nearly impossible. CPSC staff stated that little is known about many chronic illness hazards that may be associated with potentially dangerous substances, and even less information is available about which consumer products may contain these substances. Chronic illnesses are especially likely to be underestimated in CPSC’s NEISS data because they are underrepresented among emergency room visits and because product involvement is more difficult to ascertain. Similarly, consumer product involvement is very seldom recorded on death certificates in cases of chronic illnesses. CPSC’s surveillance data also give an incomplete picture of the severity of incidents. Although the data capture many relatively severe injuries—that is, those that result in death or require treatment in an emergency medical facility—data are missing for individuals who are admitted to the hospital through their physician rather than through the emergency room. Potentially less severe cases—for example, those treated in physicians’ offices, walk-in medical centers, or hospital outpatient clinics—are not represented at all in CPSC’s systematic surveillance data systems, and consequently, CPSC has no data on some consumer product problems that may result in numerous but potentially less severe injuries. Sketchy information about accident victims also limits CPSC’s ability to assess which consumer product hazards have a disproportionate impact on vulnerable populations. NEISS and death certificates provide only the 15 These figures were derived from data provided by the American Hospital Association. 16 Precise estimates on the impact of changing treatment patterns are rare, and the impact may vary among local areas. However, one research team found that because of the growth in alternative facilities in their local area, emergency rooms handled about half of outpatient injury cases—down from a previous estimate of 75 percent. See Julian A. Waller, Joan M. Skelly, and John H. Davis, “Treated Injuries in Northern Vermont,” Accident Analysis and Prevention, 27 (6) (1995), pp. 819-28. Page 18 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 age of the victim; no systematic or comprehensive information is available to determine whether a given hazard has a special impact on other vulnerable populations, such as persons with disabilities. A former commissioner told us that the lack of other demographic information such as race, income, and disability status made it difficult for her to know which subpopulations were predominantly affected by a particular hazard. Another commissioner echoed this concern, and said that such information would be useful in targeting public information campaigns on certain hazards to those groups that needed the information most. CPSC staff identified the need for additional exposure data as a major concern. However, they also told us that obtaining information on exposure to products and establishing causation requires special efforts that can be time consuming and costly. Although CPSC’s priority-setting criteria include exposure to the hazard, exposure data are generally not included in CPSC’s ongoing data collection efforts. As a result, exposure is assessed either not at all or further along in the project, precluding the use of exposure as an effective criterion for project selection. Similarly, CPSC’s emergency room and death certificate data provide little information on the circumstances surrounding the incident. As a result, CPSC staff perform follow-up investigations of selected incidents to obtain additional detail. These investigations may include detailed interviews with victims and/or witnesses, police or fire reports, photographs of the product and/or accident site, laboratory testing of the product, or recreations of the incident. As with exposure data, these investigations are not conducted for every project and are completed only after a project is well under way. Thus, assessment of causation at the project selection stage is unavoidably speculative. Agency Project Management CPSC conducts a number of projects annually, but staff were unable to Information Incomplete or provide a comprehensive list of projects the agency had worked on in the Unavailable 6-year period we examined. CPSC was also unable to verify the completeness of the project list we compiled from agency documents and interviews with staff. According to CPSC staff, internal management systems do not contain this information and such a list could be compiled only by relying on institutional memories of staff members who had been with the agency long enough to know which products the agency had addressed. Without systematic and comprehensive information on its past efforts, CPSC cannot assess whether some hazard areas have been overrepresented and whether agency resources might be more efficiently employed. Page 19 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 CPSC also lacks information on the characteristics of, resources used on, or outcomes of individual projects. CPSC’s MIS tracks contract dollars and staff time by accounting codes that cover some specific projects and general categories, such as compliance work, which are composed of numerous activities. According to agency officials, CPSC’s MIS generally cannot provide descriptive information on individual projects, such as when a project was started or concluded; the number of staff days used; what aspect of the product was addressed; whether the project originated from a petition, congressional mandate or other source; or what action was taken to address the hazard (mandatory standard, voluntary standard, or public information campaign, for example). In addition, CPSC staff told us that two separate projects involving the same or similar products at different times may be assigned the same MIS code. As a result, even if a project appears to be tracked in the MIS, reliable inferences cannot be drawn from MIS data. CPSC Has Limited Ability to CPSC’s limited data on deaths and injuries, combined with its lack of Evaluate Agency Impact information on projects, reduce the agency’s ability to evaluate the impact of its work, a process it is now required to undertake under recently passed legislation. The Results Act requires every federal agency to evaluate the effectiveness of its efforts starting in fiscal year 1999. The Results Act is aimed at increasing the investment return of tax dollars by improving agencies’ performance. Under the Results Act, an agency is to set mission-related goals and measure progress toward these goals to evaluate agency impact. CPSC has preliminarily identified results-oriented goals in four areas: (1) reducing head injuries to children, (2) reducing deaths from fires, (3) reducing deaths from carbon monoxide poisoning, and (4) reducing deaths from electrocutions. However, the limitations in CPSC’s injury and death data raise a question about how well CPSC will be able to evaluate the effectiveness of agency actions in these and other areas. CPSC uses two analytical tools—risk assessment and cost-benefit CPSC Uses analysis—to assist in making decisions on regulatory and nonregulatory COST-BENEFIT methods to address potential hazards. Risk assessment involves estimating Analysis and Risk the likelihood of an adverse event (such as injury or death). For example, CPSC estimated that the risk of death from an accident involving an Assessment, but all-terrain vehicle (ATV) was about 1 death for every 10,000 ATVs in use in Improvements Are 1994. Cost-benefit analysis details and compares the expected effects of a proposed regulation or policy, including both the positive results Needed in Data and (benefits) and the negative consequences (costs). Although cost-benefit Methodology Page 20 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 analysis may not be applicable to every decision and may not be the only factor appropriately considered in a decision, it can be a useful decision-making tool. The Congress requires CPSC to perform cost-benefit analysis before issuing certain regulations, and CPSC has conducted cost-benefit analysis for these regulations and in other situations in which it was not required. Although perfectly complete and accurate data are rarely available for any analysis, CPSC’s data are frequently inadequate to support detailed, thorough, and careful risk assessment and cost-benefit analysis. In addition, CPSC’s cost-benefit analyses are frequently not comprehensive, and the reports on these analyses are not sufficiently detailed. Improvements in the agency’s methodology and in the quality of the underlying data are necessary to ensure the clarity and accuracy of CPSC’s risk assessments and cost-benefit analyses. CPSC Performs Cost-benefit analysis can help decisionmakers by organizing and Cost-Benefit Analysis More aggregating all the relevant information to clarify the nature of the Often Than Required by trade-offs involved in a decision. Although cost-benefit analysis may not be appropriately used as the sole criterion for making a decision, a Law well-constructed cost-benefit analysis can highlight crucial factors, expose possible biases, and facilitate informed decisions even when it is impossible to measure all the potential effects of a specific regulatory proposal. The Congress has required CPSC to perform and publish a cost-benefit analysis when issuing a regulation (such as a mandatory standard or product ban) under the Consumer Product Safety Act. In addition, CPSC is also required to conduct cost-benefit analyses before issuing regulations under the authority of portions of the Federal Hazardous Substances Act (specified labeling provisions are exempt from this requirement) and the Flammable Fabrics Act.17 Because most of the agency’s projects do not involve mandatory regulation, relatively few CPSC projects conducted between January 1, 1990, and September 30, 1996, were subject to these requirements. We identified 8 cost-benefit analyses that CPSC performed in accordance with these requirements, and an additional 21 analyses it conducted in situations in which it was not required. For example, CPSC performed cost-benefit analyses in eight instances in which it was considering issuing requirements for child-resistant packaging under the Poison Prevention 17 As an independent agency, CPSC is not required to comply with Executive Order 12866, which requires federal agencies to perform a cost-benefit analysis and submit it to the Office of Management and Budget (OMB) for review whenever the agency issues a regulation that is economically significant. Page 21 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Packaging Act, which does not require cost-benefit analysis.18 CPSC frequently conducts cost-benefit analysis with respect to regulatory procedures, whether or not it is required to do so. However, a complete cost-benefit analysis is done less frequently for voluntary standards projects or information and education efforts, although some economic information may be generated to assist such projects. In addition to the complete cost-benefit analyses, we identified an additional 23 cases in which some information was provided on some economic benefits or costs. Before issuing a mandatory regulation, CPSC is required to consider the degree and nature of the risk of injury the regulation is designed to eliminate or reduce. However, CPSC usually does not conduct a formal numerical risk assessment before issuing a regulation, and the law does not require it. We found 24 risk assessments conducted by CPSC between January 1, 1990, and September 30, 1996; only 4 of these were associated with regulatory action. Agency Data Are Generally Both risk assessment and cost-benefit analysis require extensive data. Risk Insufficient to Support assessment requires information both on the adverse event and on Thorough and Detailed exposure to the precipitating circumstances. For example, when CPSC performed a risk assessment on floor furnaces, the agency estimated the Analysis number of previous injuries associated with floor furnaces and the number of floor furnaces in use.19 Similarly, when CPSC performed a risk assessment to examine the risk of contracting cancer from dioxin traces in common paper products, the agency used information from laboratory studies on dioxin’s link to cancer and also incorporated data on exposure to paper products.20 Because cost-benefit analysis includes a comprehensive delineation of the expected effects of a given proposal, a careful and thorough cost-benefit analysis will also be very data intensive and rich with detail. CPSC’s data systems are frequently unable to adequately meet the extensive demands for information posed by risk assessment and cost-benefit analysis. As a result, the agency’s estimates of risks, costs, and benefits are 18 For child-resistant packaging regulations, CPSC is legally required to consider only whether requiring child-resistant packaging is “technically feasible, practicable, and appropriate.” 19 CPSC found that the risk of injury associated with floor furnaces was small in terms of the number of incidents but relatively higher when considered in relation to the number of products in use. 20 CPSC’s study found that the risk from this type of dioxin exposure was negligible—at most, 5 cases per billion people exposed—and as a result no action was taken. Page 22 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 less accurate because they reflect the substantial limitations of the underlying data. Available information does not permit us to determine the potential impact of better data on the results of CPSC’s cost-benefit analyses and risk assessments. Some of these data weaknesses tend to make product hazard risks seem larger, and other problems tend to make the same risks appear smaller. For example, because CPSC’s data undercount the deaths and injuries associated with particular consumer products, estimates of risk—and the potential benefits of reducing that risk—appear smaller. However, CPSC’s surveillance data provide information only on whether a product was involved in an accident, not on whether the product caused the accident. At least at the initial stages of a project, this can make the risks assessed by CPSC—and the benefits of reducing those risks—appear larger. For risk assessment, CPSC must also obtain information on exposure to the hazard. For example, to assess the risk associated with aluminum ladders, CPSC obtained estimates of the number of ladders available for use and on the number of times each year the ladder was used. Obtaining exposure data presents special challenges for CPSC. Because the product definition that relates to a particular hazard is often relatively narrow, existing data sources frequently offer insufficient detail. For example, CPSC was unable to use Census sources to determine the number of saunas in the United States because saunas were included in a broader classification of products when government data were collected. In addition, CPSC staff told us that it is often difficult to find accurate information on the number of products that are in households and available for use. CPSC sometimes responds to these challenges by using mathematical modeling techniques or easier-to-obtain proxy measures (such as population) to estimate product exposure. In addition, for a few large-scale projects, CPSC has incurred the substantial expenses necessary to conduct its own detailed exposure survey. For example, CPSC conducted a survey of households that asked detailed questions on matches and disposable cigarette lighters—the number purchased, where they are generally kept, how they are used, and other details. However, for the majority of the projects we reviewed, CPSC did not gather any data on exposure. Of the 80 projects we reviewed for which briefing packages were prepared, only 26 included information on exposure to the hazard, and CPSC’s risk assessments were confined to 24 cases between 1990 and 1996—approximately 21 percent of all projects conducted over that time period. Page 23 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 CPSC Cost-Benefit The methodology used to conduct a cost-benefit analysis will frequently Analyses Are Often Not depend on the circumstances and the context of the analysis. For this Comprehensive and Not reason, there is no complete set of standards for evaluating the quality of an individual cost-benefit analysis. However, the professional literature Reported in Sufficient offers some guidance for analysts, and certain specific elements are Detail frequently mentioned as essential for cost-benefit analysis. For example, because cost-benefit analysis is meant to be a complete delineation of the expected effects of a proposed action, all potential impacts (even those that cannot be quantified) should be discussed. To ensure that the reader is able to make an informed judgment, it is important to be explicit about the underlying data, methodology, and assumptions. Accordingly, the literature suggests that all methodological choices and assumptions should be detailed, all limitations pertaining to the data should be revealed, and measures of uncertainty should be provided to allow the reader to take into account the precision of the underlying data. Similarly, the literature calls for sensitivity analysis, which enables the reader to determine which assumptions, values, and parameters of the cost-benefit analysis are most important to the conclusions. On the basis of our review of the cost-benefit literature, we developed a list of the elements that are frequently used in evaluating cost-benefit analysis. This list, and a description of all the factors we examined, is in appendix IV.21 Although we compared each of these elements with each of CPSC’s analyses, not all elements were applicable to each case. For example, in some cases, the circumstances indicated by a given element—such as reliance on statistical data—were not found, and those cases were treated as not applicable to that element. In addition, for some elements it was not always possible to determine whether CPSC’s analysis was consistent with the element. For these reasons, and to emphasize those areas that we viewed as most critical, we reported only the evaluation results that relate to key elements, applied to the majority of CPSC’s analyses, and for which a determination was possible in all or nearly all cases. Our review of all the cost-benefit analyses that CPSC conducted between January 1, 1990, and October 31, 1996, showed that for many—but not all—elements, CPSC’s analyses were not comprehensive and not reported in sufficient detail (see table 2). For example, CPSC provided descriptive information on proposals and also provided information on a variety of reasonable alternatives in almost 100 percent of cases. However, CPSC 21 These elements are also similar to guidance OMB issued in January 1996 for preparing economic analyses of significant regulatory actions. Page 24 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 analyses generally did not provide measures of uncertainty for the underlying data. Estimates derived from samples are subject to sampling error, which can be especially large when the estimates are projected from relatively fewer cases. In only 17 percent of its analyses did CPSC provide any statistical information on the precision of the underlying estimates. Similarly, when estimates are based on a relatively small sample size, projections are generally not considered reliable. CPSC analysts cautioned the reader against making conclusions based on small sample data only 45 percent of the time. In addition, some of CPSC’s data sets have a known upward or downward bias because of the way the data were constructed. For example, CPSC’s estimates of deaths based on CPSC’s death certificate database will be understated, and when estimates of incidents are based only on investigated or reported cases (such as cases reported to CPSC’s hot line), two potential biases are likely to be introduced into the analysis: (1) the estimates are likely to be biased downward by nonreporting and (2) the incidents reported tend to be the more severe ones. In only 53 percent of applicable cases did CPSC’s analysis inform the reader of known limitations inherent in the data being used for cost-benefit analysis. Table 2: Evaluation of CPSC Analyses Shows Inconsistencies With Several Percentage of Evaluation Elements CPSC’s analyses that were consistent Evaluation element with this element Provided descriptive information about a well-defined proposal 98 Addressed multiple alternatives 95 Reported measures of precision for underlying data 17 Cautioned reader about making inferences from data with a small sample size 45 Reported known biases in underlying data 53 Provided any sensitivity analysis information 26 Included all important categories of benefits and costs 54 Considered “risk-risk” trade-offsa 49 a A “risk-risk” trade-off refers to an action to decrease a hazard’s risk that unintentionally increases that or another risk. We identified several other areas in which CPSC analyses could benefit from improvement. For example, researchers agree that sensitivity analysis—a technique that enables the reader to determine which assumptions, data limitations, or parameters are most important to the conclusions—should be incorporated in cost-benefit analyses. CPSC usually did not provide sensitivity analysis information. For example, agency Page 25 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 briefing packages did not include any information on how CPSC’s injury cost estimates were derived or what factors were the largest components of injury costs. CPSC applies a statistical model to injury estimates to derive a figure for injury cost. The model that computed injury cost estimates accounts for a number of components, including medical costs, forgone wages, and pain and suffering. With only one exception, CPSC briefing packages provided only the total cost, without any information on the derivation of those costs or the individual components. In addition, CPSC provided only an average injury cost, not a range of injury cost estimates. For situations in which injuries differ in severity, or for projects in which severity is probably overstated or understated in the data, the reader would find such information useful. Forty-six percent of CPSC analyses did not consider the full range of costs and benefits likely to result from regulation. For example, CPSC analysts frequently omitted mentioning intangible costs and/or benefits (costs or benefits that are difficult to quantify, such as loss of consumer enjoyment) or potential indirect effects (such as changes in the prices of related goods). In addition, CPSC frequently excluded risk-risk considerations from its evaluation of the costs and benefits of potential actions. Sometimes actions taken to reduce one risk can have the unintended effect of increasing that or another risk. Individuals may take more or fewer precautions in response to a change in a product’s safety features, and this behavior can result in an increase in the risk the intervention was designed to mitigate. For example, in establishing a standard for child-resistant packaging that was also “senior-friendly,” CPSC considered that because child-resistant medicine bottles can be difficult to open, a grandparent may leave the cap off the bottle, creating an even greater risk than would be the case with the original cap. Although CPSC considered such factors in some cases, only 49 percent of its analyses reflected potential risk-risk trade-offs. CPSC has not established internal procedures that require analysts to conduct comprehensive analyses and report them in sufficient detail. For example, according to CPSC staff, the agency has little written guidance about what factors should be included in cost-benefit analyses, what methodology should be used to incorporate these factors, and how the results should be presented. Staff also told us that CPSC analyses are not generally subject to external peer review. Such reviews can serve as an important mechanism for enhancing the quality and credibility of the analyses that are used to help make key agency decisions. Page 26 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 To help minimize the possibility that a product might be unfairly CPSC Has Established disparaged, in section 6(b) of the Consumer Product Safety Act the Procedures for Congress imposed restrictions on the disclosure of manufacturer-specific Complying With information by CPSC.22 Before CPSC can disclose any information that identifies a manufacturer,23 the agency must Statutory Requirements for • take “reasonable steps” to verify the accuracy of the information and to ensure that disclosure is fair, Releasing • notify the manufacturer that the information is subject to release, and Manufacturer-Specific • provide the manufacturer with an opportunity to comment on the Information information. If the manufacturer requests that its comments be included in CPSC’s disclosure of the information, CPSC can release the information only if accompanied by the manufacturer’s comments. If the manufacturer objects to the release even if its comments are included, it can challenge CPSC in U.S. district court to block disclosure. These restrictions on the release of information apply not only to information the agency issues on its own—such as a press release—but also to information disclosed in response to a request under FOIA. In addition, section 6(b) also requires CPSC to establish procedures to ensure that releases of information that reflect on the safety of a consumer product or class of products are accurate and not misleading, regardless of whether the information disclosed identifies a specific manufacturer. CPSC has established procedures to implement these requirements, including requiring technical staff to “sign off” on information releases and notifying manufacturers. Evidence from several sources—industry sources, published legal decisions, and agency retractions—suggests that CPSC has complied with its statutory requirements. CPSC staff and commissioners, industry representatives, and consumer advocates expressed a wide variety of opinions on the effectiveness of these requirements, and some individuals favored specific changes. 22 An exception to these restrictions is given if CPSC has declared that the product is an “imminent hazard” under section 12 of the Consumer Product Safety Act. 23 These restrictions also apply even if the manufacturer is not named but the information would allow the reader to readily identify the manufacturer from the context. For example, if there is only one manufacturer of a product that is identified in the information, the information may be subject to restriction even if the manufacturer’s name is not given. Page 27 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 CPSC Has Established Part of CPSC’s mission is to provide the public with information to help Procedures for Verifying individuals use consumer products safely. CPSC disseminates information and Clearing Information through its own initiatives and also in response to requests from the public. For example, CPSC informs both consumers and businesses about for Release product hazards through product recall notices, provision of information at trade shows and special events, and a telephone hot line and Internet site. In addition, CPSC responds to thousands of telephone and written requests for information each year. CPSC’s mission and its responsibility under FOIA require the agency to disseminate a great deal of information. However, because much of this information is about specific products or manufacturers, CPSC’s information disclosure is often restricted under section 6(b). In its regulation implementing section 6(b), CPSC established several measures designed to ensure compliance with the statutory requirements. These measurers include obtaining written verification from consumers of the information they report to the agency, notifying manufacturers by certified mail when manufacturer-specific information is requested, and giving manufacturers the option of having their comments published along with any information being disclosed. CPSC’s procedures outline several steps to verify all information before it is released. For example, CPSC checks each report the agency receives from consumers about incidents involving potentially hazardous products to ensure that CPSC’s records accurately reflect the consumer’s version of the incident. Agency procedures require staff to send a written description of each incident back to the person who reported it with a request that he or she review it and state if any information needs to be corrected or supplemented.24 The commission staff review each of these incident reports for discrepancies or any obvious inaccuracies. Once they have been checked and confirmed with the consumer, incident reports are made available to the public upon request. If the reports contain information that would identify a specific manufacturer, they are subject to 6(b) requirements regarding disclosure. CPSC also investigates events surrounding selected product-related injuries or incidents. Investigation reports provide details about incident sequence, human behavior factors, and product involvement. The reports generally contain the consumer’s version of what happened and the observations of witnesses, fire and police investigators, and others. Investigations may 24 The cover letter that accompanies the incident report also asks recipients if they would like to have their names withheld if information about the incident is made public. Page 28 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 also include follow-up inspections at retail stores or service centers. However, neither investigations nor incident reports include the manufacturer’s view of the incident. Its point of view may be expressed in comments it submits before the report is released. Like incident reports, investigation files are available to the public upon request and are subject to 6(b) requirements. CPSC has issued clearance procedures to cover situations in which commissioners or staff initiate public disclosures—for example, when the Commission publishes the results of agency research. These procedures are intended to verify any information—oral or written—released by the Commission, regardless of whether the information identifies a manufacturer. Under CPSC’s guidelines, each assistant or associate executive director whose area of responsibility is involved must review the information and indicate approval for the release in writing.25 After all other review has been completed, the Office of the General Counsel must also review and approve the release. Press releases with respect to product recalls are written and issued jointly by CPSC and the affected manufacturer. In addition, CPSC’s clearance procedures for press releases state that final clearance must be obtained from the Office of the Chairman of the Commission.26 In addition, CPSC staff told us that the current chairman’s policy of coordinating media inquiries through the Office of Public Affairs is intended to ensure that information provided is in compliance with section 6(b). CPSC has also established procedures to implement the notification and comment provisions of section 6(b). Before CPSC releases information in response to an FOIA request, an information specialist determines whether a manufacturer could be readily identified. CPSC staff said that agency policy is to clearly and narrowly identify hazardous products (including by manufacturer) whenever possible, in order to prevent the person receiving the information from confusing safe products with unsafe products of the same type. However, if an information request is broad, like “all bicycle accidents,” names of manufacturers are removed before the information is released, according to CPSC staff. If the requested information could identify a manufacturer, then staff review the information for appropriate exemptions (such as trade secrets), and delete portions as appropriate. The manufacturer is given 20 calendar days in which to review and 25 For a description of CPSC’s directorates, see app. II. 26 An exception is made for those hazards considered by the Compliance staff as most serious, for which clearance must be obtained from a majority of the commissioners. Page 29 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 comment on a summary of the information CPSC plans to release.27 Because CPSC often receives multiple requests for the same information, the agency informs manufacturers that it will not send them copies of subsequent requests for the same information unless specifically requested to do so. However, according to CPSC staff, more than 80 percent of the manufacturers that submit 6(b) comments routinely request such notification. In calendar year 1993 (the most recent year for which data were available), CPSC sent out 487 notices to manufacturers and received 154 responses (32 percent). Twenty-five manufacturers (5 percent) contested the accuracy of the information or claimed that the proposed disclosure would be unfair. If a manufacturer fails to comment, the information can be released 30 days from the date CPSC notified the manufacturer. After taking the manufacturer’s comments into account, CPSC may decide to disclose incident information despite the firm’s objection if, for example, the comments lack specific information to support a claim of inaccuracy or a request for confidentiality. If CPSC chooses to disclose information over a manufacturer’s objection, it must release the manufacturer’s comments along with the other information, unless the manufacturer requests otherwise. In addition, if CPSC decides to release information and the manufacturer objects, the manufacturer has 10 working days to go to court and seek to enjoin CPSC from disclosing the information. Manufacturers have sued CPSC to prohibit disclosure of records only 11 times since the agency was founded, and CPSC was prohibited from releasing the information in 2 of these cases.28 Evidence Suggests That Information from three sources of evidence—industry, published legal CPSC Complies With Legal cases, and data on retractions—suggest that CPSC complies with its Requirements Regarding statutory requirements concerning information release. Industry sources, even those otherwise critical of the agency, told us that CPSC generally the Release of Information does a good job of keeping proprietary information confidential as required by law. Our review of published legal decisions found no rulings that CPSC violated its statutory requirements concerning the release of information. Retractions by CPSC are also rare. If CPSC finds that it has disclosed inaccurate or misleading information that reflects adversely on any consumer product or class of consumer products or on any 27 Since small companies may lack the technical ability to comment, CPSC staff told us that they will sometimes help small businesses to formulate their comments. 28 Currently, Daisy Manufacturing, Inc. v. CPSC is on appeal, after a judge ruled in CPSC’s favor in U.S. district court. Page 30 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 manufacturer, it must publish a retraction. Any retraction must be made in the same manner in which CPSC released the original information. According to CPSC, it has published only three such retractions. Two of these retractions, in 1984 and 1994, were made in response to requests from firms. A third retraction, in 1990, was issued after CPSC discovered that a report in its public reading room had mistakenly included inaccurate information. Industry Observers, Industry observers, CPSC staff, and consumer groups expressed a wide Consumer Groups, and range of opinions on the effectiveness of section 6(b). In response to our Others Suggested Changes inquiries, some CPSC commissioners and former commissioners said that these restrictions serve a useful purpose and should not be changed. to 6(b) Requirements However, CPSC’s current chairman, industry and advocacy group representatives, and others expressed dissatisfaction with 6(b), and some of these suggested possible changes. Although these individuals raised issues about the extent of the protection afforded to manufacturers and the resources necessary to ensure compliance, we did not assess whether the specific suggestions were necessary or feasible. CPSC’s chairman, other CPSC officials, former commissioners, and the representative of a consumer advocacy group stated that compliance with 6(b) is costly for CPSC and delays the agency in getting information out to the public. Although CPSC has not estimated the cost of complying with 6(b), agency staff told us that it takes much more staff time to respond to FOIA requests that come under 6(b) than it does to respond to FOIA requests that do not involve company names. To reduce the burden of complying with these requirements, CPSC staff have suggested that the notification requirement that gives manufacturers 20 days in which to comment should apply only the first time an item is released. Some have suggested that instead of requiring CPSC to verify information from consumer complaints, the agency should be allowed to issue such information with an explicit disclaimer that CPSC has not taken a position on the validity of the consumer’s report. Instead of reducing CPSC’s verification requirements, some industry representatives suggested expanding them. These manufacturers stated that before CPSC releases incident information, it should substantiate the information rather than relying on a consumer’s testimony. Industry representatives stated—and CPSC staff confirmed—that many of the requests for CPSC information come from attorneys for plaintiffs in product liability suits. As a result, some industry representatives expressed Page 31 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 concern that unsubstantiated consumer complaints could be used against them in product liability litigation. They suggested that 6(b) should require CPSC to substantiate all incident reports by investigating them before they can be disclosed instead of merely checking with the consumer as it does now. However, CPSC officials told us that investigations—which are time consuming and costly—can be conducted only on a small proportion of specially selected cases because of limited resources. Industry representatives also said that the current restrictions do not provide sufficient protection when information is released on product groups instead of on the products of an individual manufacturer. Several industry representatives expressed concern that producers of safer products may be unfairly maligned when CPSC releases information about a group of products, only some of which may be associated with a safety problem. According to some of these industry representatives, CPSC should extend protection to product groups similar to the safeguards manufacturers receive under 6(b). Retailers’ representatives also suggested specific changes to CPSC’s information release requirements. They said that retailers do not receive timely notice of recalls because CPSC has interpreted the law to prohibit advance notification of retailers. Consequently, the retailers said that they sometimes receive notice of recalls at the same time as their customers and have no time to prepare. For example, when consumers come in with recalled products, the retailer may not yet know whether the manufacturer has agreed to replace the product, refund the purchase price, or provide some other remedy. Retailers’ representatives suggested amending 6(b) to give 5 business days’ advance notice to retailers before the public announcement of a recall. CPSC officials said that typically manufacturers are and should be the ones to contact the retailers and make all arrangements for a recall. Although they disagreed on the need for a statutory change, both CPSC staff and a major retailers’ association said that they were trying to work out a more satisfactory arrangement. CPSC’s current data provide insufficient information to monitor ongoing Conclusions projects and to determine whether potential projects adhere to the agency’s selection criteria. Moreover, inadequate agency data often prevent CPSC from conducting risk assessments on projects, potentially limiting the agency’s ability to target resources to the hazards presenting the greatest risks. The lack of sufficient data, combined with methodological problems, also makes CPSC’s cost-benefit analyses less Page 32 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 useful than they could be. With more detailed information on both internal resources and external product hazards, CPSC would be better able to assure the Congress and taxpayers that its resources are expended wisely. We identified several key areas where CPSC management could improve its collection and analysis of external data. For example, CPSC would be better able to make informed decisions on potential agency projects if it had additional statistically reliable and timely data in several areas, including (1) injuries treated outside of hospital emergency rooms; (2) exposure to consumer products and product-related hazards; (3) chronic illnesses related to consumer products; and (4) hazards that disproportionately affect certain vulnerable populations, such as low-income individuals and consumers with disabilities. In addition, project selection and implementation could be improved if commissioners and staff had tracking information on CPSC projects, such as starting and ending dates, project origin, project costs (including staff days and contract costs), and agency actions taken to address the potential hazard. Such information could assist the commissioners in monitoring ongoing projects, targeting new efforts on the basis of previous work, and assessing the allocation of resources across current projects and among major hazard areas. CPSC could also benefit from an improved methodology for cost-benefit analysis. A stronger methodological base for CPSC’s cost-benefit analyses, including more complete documentation, would promote sound regulatory decision-making and could improve the quality of the input and comment CPSC receives during the regulatory process. Without these improved data, CPSC will remain unable to accurately apply measurable criteria in choosing projects or to rigorously assess relative risks among potential hazards. Some of this necessary information—the need for more representative and complete injury and exposure data, for example—may require a significant investment of resources, so CPSC may need to prioritize these data needs. In doing so, is important for CPSC to draw on the insight of individuals outside the agency to ensure that all available alternatives for obtaining these data are explored. Some of the other information CPSC needs, however, could be compiled internally at relatively little additional cost and effort. For example, more detailed information on individual projects could be collected within the agency. Similarly, the methodological problems we identified in CPSC’s cost-benefit analysis could be remedied without additional data. Page 33 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 We recommend that the Chairman of CPSC take the following actions: Recommendations • Improve the quality of CPSC’s injury, death, and exposure data by consulting with experts both within and outside the agency to (1) prioritize CPSC’s needs for additional statistically valid surveillance data on injuries and deaths related to consumer products and on exposure to consumer products and product-related hazards, (2) investigate the feasibility and cost of alternative means of obtaining these data, and (3) design data systems to collect and analyze this information. • Direct agency staff to develop and implement a project management tracking system to compile information on current agency projects. For each project, such a system should include, at a minimum, a description of the hazard addressed, start and end dates, project origin, and major agency action resulting from it. • Direct agency staff to develop and implement procedures to ensure that all cost-benefit analyses performed on behalf of CPSC are comprehensive and reported in sufficient detail, including providing measures of precision for underlying data, incorporating information on all important costs and benefits, and performing sensitivity analysis. We received two separate sets of comments from CPSC’s Agency Comments commissioners—one from Chairman Brown and Commissioner Moore and one from Commissioner Gall. CPSC staff also submitted some technical comments, which we incorporated in the report as appropriate. Chairman Brown and Commissioner Moore stated that they are considering our recommendations and that in some respects our recommendations parallel efforts already under way at the Commission. However, they disagreed with some of the specific findings of our report. They stated that CPSC’s actions are based on solid injury and death estimates and that CPSC (1) employs sound economic analyses that are appropriate for the circumstances, (2) tracks projects to monitor the progress of its work, and (3) has been successful in dramatically reducing the threat to consumers from unsafe products. We concluded that CPSC’s death and injury data are generally insufficient to support the agency’s project selection process for two reasons: (1) CPSC has little or no data on several project selection criteria and (2) CPSC’s data on its other project selection criteria exhibit significant gaps. Similarly, our finding that CPSC’s cost-benefit analyses were not comprehensive and not reported in sufficient detail supports our conclusion that these analyses were less useful than they could have been in the agency’s decision-making process. Page 34 GAO/HEHS-97-147 Consumer Product Safety Commission B-274042 Because CPSC’s current management information system operates at a very high level of generality and (according to CPSC staff) does not produce consistent, reliable information, we recommend that the agency implement an improved tracking system that would provide enough information to monitor the projects selected and the resources spent for each hazard. We did not review whether CPSC’s actions had been successful in reducing the number of injuries and deaths associated with consumer products. The full text of Chairman Brown and Commissioner Moore’s detailed comments and our response are in appendix VI. Commissioner Gall stated in her comments that she is looking forward to implementing many of the reforms we recommended. She also stated that she agrees with our conclusion that CPSC could improve the way it gathers information, including reassessing the need for injury data gathered outside of hospital emergency rooms. Commissioner Gall also agreed that CPSC needs an improved system to track ongoing activities or projects and that sensitivity analysis, measures of uncertainty, and risk-risk analysis should be incorporated into CPSC analyses. However, she also commented that additional GAO analysis of the implications of inadequate data could have been helpful. Unfortunately, available information does not permit us to determine the impact of better-quality data on the decisions made by CPSC. In addition, Commissioner Gall said that she believes the Compliance function of CPSC also warrants further review. Such a review is outside the scope of this report. Commissioner Gall’s detailed comments and our response are in appendix VII. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies to the commissioners of the Consumer Product Safety Commission and make copies available to others upon request. If you or your staff have any questions about this report, please contact me at (202) 512-7014. Major contributors are listed in appendix VIII. Carlotta C. Joyner Director, Education and Employment Issues Page 35 GAO/HEHS-97-147 Consumer Product Safety Commission Contents Letter 1 Appendix I 40 Scope and Methodology Appendix II 43 Agency Organization and Structure Appendix III 47 CPSC Projects, 1990-96 Appendix IV 59 Background 62 A Review of the Conceptual, Analytical, and Technical Issues Arise When 64 Literature on Applying COST-BENEFIT Analysis COST-BENEFIT Analysis Appendix V 76 CPSC Relies Heavily on Hospital Emergency Room Reports to 76 Death, Injury, and Obtain Estimates of Injuries Incident Data Sources CPSC Obtains Most of Its Fatality Information From Selected 79 Death Certificates Used by CPSC CPSC Investigates Selected Incidents to Obtain Detailed 81 Information About the Causes of Deaths and Injuries CPSC Uses Modeling Techniques and Special Surveys to Estimate 82 Exposure Information CPSC Obtains Some Information Under Reporting Requirements 83 for Manufacturers and Other Businesses Page 36 GAO/HEHS-97-147 Consumer Product Safety Commission Contents Appendix VI 85 GAO Comments 102 Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Appendix VII 106 GAO Comments 110 Comments From Commissioner Gall, Consumer Product Safety Commission, and GAO’S Evaluation Appendix VIII 111 GAO Contacts and Staff Acknowledgments Bibliography 112 Tables Table 1: CPSC Regulatory Priority-Setting Criteria and Systematic 16 Information Available Table 2: Evaluation of CPSC Analyses Shows Inconsistencies 25 With Several Evaluation Elements Table III.1: CPSC Projects, 1990-96 48 Table IV.1: Frequently Used Elements of Evaluation of 60 Cost-Benefit Analysis Figures Figure 1: CPSC Funding, in Inflation-Adjusted Dollars, 1974-96 5 Figure 2: Origination of CPSC Projects, January 1, 9 1990-September 30, 1996 Page 37 GAO/HEHS-97-147 Consumer Product Safety Commission Contents Figure 3: Types of Death and Injury Data Covered by CPSC’s 15 Systematic Surveillance Information Figure II.1: CPSC Organization Chart 46 Abbreviations AAPCC American Association of Poison Control Centers ANPR Advance Notice of Proposed Rulemaking ATV all-terrain vehicle CO carbon monoxide CPSC Consumer Product Safety Commission FOIA Freedon of Information Act HMO health maintenance organization IPII Injury or Potential Injury Incident LP liquified petroleum MECAP Medical Examiners’ and Coroners’ Alert Program MIS management information system NEISS National Electronic Injury Surveillance System NFIRS National Fire Reporting System NFPA National Fire Protection Association OMB Office of Management and Budget Page 38 GAO/HEHS-97-147 Consumer Product Safety Commission Page 39 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix I Scope and Methodology This report provides information on project selection, analytical research, and information release procedures at the Consumer Product Safety Commission (CPSC), which we gathered from a variety of sources inside and outside the agency. For example, we interviewed current CPSC staff, including analysts and line managers, to obtain information about CPSC’s data and procedures. We also interviewed the three current CPSC commissioners and four former commissioners. In addition, we interviewed representatives of manufacturers, retailers, trade associations, consumer groups, and academic and other experts to obtain their perspectives on CPSC’s activities. We reviewed the legislation governing CPSC’s activities and the associated regulations. We also reviewed statements by public and private organizations and the academic research literature dealing with consumer protection issues and the technical literature on cost-benefit analysis and risk assessment. We gathered extensive information from CPSC documents. We reviewed every agency budget request, operating plan, mid-year review, and regulatory agenda from 1990 to 1996. We reviewed all CPSC’s annual reports from fiscal years 1986 to 1996. We also obtained all CPSC Federal Register notices from January 1993 to May 1996 and selected Federal Register notices from January 1990 to December 1992. We reviewed all CPSC press releases from September 1992 to February 1996. We also reviewed 644 CPSC briefing packages prepared from January 1, 1990, to September 30, 1996, including all briefing packages that concerned a specific, identifiable potential product hazard under consideration for regulatory, voluntary, or information-gathering activities. We excluded matters exclusively related to compliance with and enforcement of existing standards, civil penalties, and internal management issues as well as other items not related to a specific potential product hazard. In addition, we reviewed other agency documents, including information-clearing procedures, documentation of data systems, consumer education materials, internal memorandums and correspondence, and selected documents downloaded from CPSC’s Internet site. In describing CPSC’s project selection, we drew on our interviews with commissioners, former commissioners, and agency staff, and we reviewed the project selection criteria in CPSC’s regulations. Although we described the criteria used by the agency, we did not make judgments about the merit of individual criteria or about whether any criterion was appropriately applied in any specific case. We compiled a list of agency projects on the basis of the documents we reviewed. We included not only major regulatory efforts but also smaller-scale projects. We gathered Page 40 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix I Scope and Methodology information on the characteristics of these projects; for example, we obtained information on how the projects originated, what action CPSC took, and (for most) how many deaths and injuries were associated with the hazard. CPSC officials examined our list and provided some information that was not readily available in agency documents. However, because CPSC’s internal data management system does not track agency projects, neither we nor CPSC were able to verify the accuracy and comprehensiveness of this information. We also reviewed agency documentation and the technical literature and interviewed outside experts to obtain information on CPSC’s data-gathering systems. To examine CPSC’s cost benefit analysis, we reviewed the technical literature and developed a set of objective evaluation questions to elicit descriptions of and to evaluate the analytical work. These evaluation questions were designed to indicate only whether CPSC’s cost benefit analyses were consistent with elements that are commonly used to evaluate whether cost-benefit analyses are comprehensive and reported in sufficient detail. Our review did not assess whether CPSC’s analyses were the best that could be done on any particular topic. We reviewed these evaluation questions with two leading experts in the field of risk assessment and cost-benefit analysis: Professor John Mendeloff of the University of Pittsburgh and Professor John Graham of Harvard University. Then we examined all CPSC’s cost-benefit analyses from January 1990 through September 1996 to see how they measured up against these evaluation questions. In making these assessments, we reviewed all the information available to the commissioners in the written record; that is, we did not confine our analysis to the portion of the briefing packages that dealt explicitly with cost-benefit analysis or risk assessment, and we examined all the briefing packages that pertained to that particular project. Although we evaluated CPSC’s cost-benefit analyses using a wide-ranging set of evaluation questions, we reported results of our analysis only for those questions where: (1) the question was applicable to the majority of the CPSC analyses we reviewed; and (2) we were able to make a determination of whether the analyses were consistent with the evaluation question in applicable cases. To review CPSC’s information release process, we reviewed internal agency procedures, discussed the information release requirements with agency officials and industry representatives, and reviewed the relevant legal cases. We provided information on CPSC’s information release procedures but did not audit whether CPSC complied with these procedures in releasing information. Instead, we relied on other sources—industry Page 41 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix I Scope and Methodology sources, published legal decisions, and retractions—to assess whether this readily available evidence suggested that CPSC had violated its statutory requirements in releasing manufacturer-specific information. Our review was conducted between August 1996 and May 1997 in accordance with generally accepted government auditing standards. Page 42 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix II Agency Organization and Structure The Consumer Product Safety Act (P.L. 92-573) provides for the appointment of five commissioners by the President of the United States for staggered 7-year terms. Not more than three of the commissioners may be affiliated with the same political party. The President appoints the commissioners, subject to Senate confirmation. However, the President cannot directly overrule an agency decision or fire a commissioner for making an unpopular decision. Since 1986, there have been no more than three commissioners at one time. In fulfilling the provisions of the government in the Sunshine Act (P.L. 94-409),29 in general, commissioners must open to the public any meeting held for the purpose of disposing of CPSC business—that is, if any two of the three commissioners want to discuss agency business, generally public notice must first be given. On a daily basis, communication among CPSC offices takes place at the staff level. The chairman, the principal executive officer of the Commission, directs all executive and administrative functions of the agency. The chairman oversees the appointment and supervision of all agency personnel except those employed in the immediate offices of other commissioners. CPSC annually elects a vice chairman to act in the absence or disability of the chairman or in case of a vacancy in the office of chairman. As figure II.1 shows, six offices report directly to the chairman: • the Office of the Secretary manages the agency’s records, publishes CPSC’s public meetings calendar, and administers the requirements of the Freedom of Information Act; • the Office of Congressional Relations serves as CPSC’s liaison with the Congress; • the Office of General Counsel provides advice and counsel to the agency on legal matters; • the Office of the Inspector General is an independent office that undertakes activities to prevent and detect waste, fraud, and abuse; and • the Office of Equal Opportunity and Minority Enterprise monitors compliance with equal opportunity employment laws. The agency’s executive director, who is appointed by and reports directly to the chairman, is responsible for overseeing the day-to-day management of the agency. The executive director’s office also houses the agency’s small business ombudsman, who acts as a liaison to small businesses, 29 The purpose of the Sunshine Act is to provide the public with information regarding the decision making processes of the federal government while protecting both the rights of individuals and the ability of the government to carry out its responsibilities. Page 43 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix II Agency Organization and Structure providing technical assistance concerning CPSC programs and regulations, among other things. The executive director has line authority over nine operating directorates: • The Office of Compliance is responsible for compliance with and administrative enforcement of CPSC regulations, including product standards. The office initiates investigations on safety hazards of products already in the consumer marketplace or being offered for import. It negotiates and subsequently monitors corrective action plans designed to give public notice of hazards and recall defective or noncomplying products. • The Office of Hazard Identification and Reduction manages many of CPSC’s activities that involve identifying, examining, and remedying new product hazards. The office’s responsibilities include collecting and analyzing data to identify hazards and hazard patterns, carrying out CPSC’s regulatory and voluntary standards development projects, and coordinating international activities related to consumer product safety. Serving in various groups under the director of hazard identification and reduction are the epidemiologists and statisticians, who provide information on particular products; engineers and human factors specialists, who help develop design remedies for product hazards; and economists, who provide market information and perform cost-benefit analysis on commission projects. • The Office of Information and Public Affairs is CPSC’s touchstone with consumers and the media. It prepares and publishes brochures, booklets, fact sheets, and safety alerts providing safety information to consumers on products used in the home environment. The Office of Information and Public Affairs also handles requests from the media for information and access to CPSC staff and prepares press releases announcing CPSC actions or decisions. • The Office of Information Services manages the agency’s toll-free hot line, Internet, and fax-on-demand services. Within the Office of Information Services, CPSC’s Information Clearinghouse provides summary information on product-related injuries in response to requests from the public.30 • The Office of the Budget is responsible for overseeing the development of CPSC’s budget. • The Office of Human Resources Management provides support to the agency in recruitment and placement, position classification, training, and other personnel areas. 30 If such a request can not be filled without identifying the manufacturer, however, the request is sent to the freedom of information officer for processing. Page 44 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix II Agency Organization and Structure • The Office of Planning and Evaluation assists with long-term planning efforts and manages and ensures agency compliance with paperwork reduction regulations. In addition, the Office of Planning and Evaluation is currently preparing for the implementation of the Government Performance and Results Act (the Results Act) and reviewing the effectiveness of the agency’s outreach efforts. • The Directorate for Field Operations coordinates the activities of CPSC’s 128 field staff located in 38 cities across the country. CPSC field staff carry out a wide range of agency activities, including conducting investigations of injury incidents; acting as liaisons with state and local organizations; working with the local press to support consumer education campaigns; and inspecting manufacturers, importers, distributors, and retailers to ensure compliance with safety regulations and standards. To complement the efforts of the field staff at the local level, CPSC contracts some product safety work to state and local entities; commissions state and local officials to function as officials of CPSC for the purpose of conducting investigations, inspections, recalls, and sample collections; and transmits information on CPSC programs and activities to states. • The Directorate for Administration is responsible for executing general administrative policies. Page 45 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix II Agency Organization and Structure Figure II.1: CPSC Organization Chart Page 46 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 CPSC has addressed many different product hazards. We identified 115 projects the agency worked on between January 1, 1990, and September 30, 1996. Because CPSC does not maintain a list of projects it has worked on or any project characteristics, such as project origin or result, we used various agency documents, such as budget requests, annual operating plans, regulatory agendas, and project briefing documents, to compile our list. We attempted to determine how each project came to the attention of the agency, for example, through a petition, congressional direction, or internal initiation. This was not always clear. For example, in three cases, a petition was submitted and subsequently denied, but the issue was later mandated by the Congress as a project the agency must address. For example, a petition concerning the safety of bicycle helmets submitted in 1990 was denied. In 1994, the Congress passed the Children’s Bicycle Helmet Safety Act of 1994, which requires CPSC to establish a mandatory standard. We counted such projects as originating through the Congress. In other cases, we categorized projects on the basis of information in CPSC documents. We also attempted to identify the most significant action that resulted from the agency’s efforts. Some projects were ongoing for several years and involved more than one agency action, such as information campaigns conducted in conjunction with voluntary standards efforts. For example, agency work on carbon monoxide (CO) detectors resulted in a voluntary standard in 1992, a “CO Awareness Week” information campaign in 1994, and public hearings on the issue in 1996. In cases such as these, we attempted to identify the activity with the greatest long-lasting impact; usually this was a voluntary or mandatory standard. Also, projects classified as “voluntary standard activity” include those for which a new voluntary standard was created, an existing voluntary standard was revised, or staff are currently participating in voluntary standard activities. In addition, we identified whether a risk assessment or cost-benefit analysis was completed for a given project, using CPSC briefing packages. We defined a formal risk assessment to include only those cases for which a numerical estimate of unit risk was calculated; we defined a complete cost-benefit analysis to include those projects for which both economic costs and benefits were explicitly compared, even if quantitative estimates were not made for all economic factors considered. Some risk-related or economic information was provided for many of the projects for which no formal, complete risk assessment or cost-benefit analysis was performed. Page 47 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Table III.1: CPSC Projects, 1990-96 Origin Initiated by the Initiated by Product Congress Petition agency Fire/gas codes and standards Camping heaters x Carbon monoxide detectors x Children’s sleep-wear flammability x Cigarette lighters x Cigarette safety x Elderly sleepwear flammability x Fire safety devices x c Fireworks, fuse-burn time x Fireworks, large and small reloadable shell x Fireworks, multiple tube mine and shell x Flexible gas connectors x Floor furnaces x Fuel gas detectors x Gas control valves, automatic x Gas furnaces x Gas grills, 20-lb. systems x Gas systems, over-pressurization x Heaters, unvented gas space heaters x Liquified petroleum (LP) gas odorant fade x Mattress/bedding fires x Multipurpose lighters x Range/oven fires x Smoke detectors x Upholstered furniture x Water heaters x Sports and Recreation All-terrain vehicles (ATV) x g Baseball injuries x Baseball bases x Page 48 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Agency action Analytical efforts Mandatory Voluntary Information and Cost-benefit Risk assessment Ban standard standard activitya education Otherb analysis done done x x x x x x x x x x x x x x x x x x x x x x x x x x x x xd x x x xe x x xf x x x (continued) Page 49 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Origin Initiated by the Initiated by Product Congress Petition agency Baseball, chest protectors for young batters x Baseball, face-guards on helmets x Baseball, safety baseballs (soft) x Bicycles x Bicycle reflectors x Bicycle/recreation helmets xh In-line skates x Model rocket motors x Pools, barriers for swimming pools and spas x Pools, diving injuries x Pools, swimming pool covers x Saunas x Soccer injuries x Soccer goals x Spas and hot tubs x Electrical/power codes and standards Garage door openers x Go-carts x Hair dryers x Heaters, portable electric x Heat tapes x Holiday lights x Home electrical system fires x Ladders, portable aluminum x Lamps, portable x Microwave ovens x Mowers, riding x Mowers, walk-behind x Receptacle outlets x Shock protection devices (GFCI) x Page 50 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Agency action Analytical efforts Mandatory Voluntary Information and Cost-benefit Risk assessment Ban standard standard activitya education Otherb analysis done done x x x x x x x x x x x x x x x x x x x x x xi x x x x x x x x x x x x x x x x x x x (continued) Page 51 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Origin Initiated by the Initiated by Product Congress Petition agency Thermoplastics in electrical products x Children’s products Baby bath seats x Baby walkers x Bean bag chairs x Bunk beds x Child restraints on grocery carts x Choking hazards—balloons, balls, marbles, x small figures, pom-poms Clacker balls x Crib corner post extensions x Cribs, nonfull size x Crib toys x Drawstrings on children’s clothing x Electronic video games x 5-gallon buckets x Furniture tipover x High chairs x Infant cushions x Infant suffocation x Lead in paint x Playground equipment, home x Playground equipment, public x Playground equipment, soft x Playground surfacing x Play yards x Strollers x Toddler beds x Window blind cords x Window guards x Chemical/poison prevention Page 52 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Agency action Analytical efforts Mandatory Voluntary Information and Cost-benefit Risk assessment Ban standard standard activitya education Otherb analysis done done x x xd x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x xj x x x x x x x (continued) Page 53 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Origin Initiated by the Initiated by Product Congress Petition agency Acetone in 1-gallon containers xk Architectural glazing x Art materials x Charcoal labeling x Child-resistant, adult-friendly packaging of drugs x Child-resistant packaging of certain dietary products with iron powders x Child-resistant packaging of dibucaine x Child-resistant packaging of ibuprofen x Child-resistant packaging of drugs with loperamide x Child-resistant packaging of glue removers with acetonitrile xl Child-resistant packaging of hair wave neutralizers xl Child-resistant packaging of lidocaine x Child-resistant packaging of mouthwashes with ethanol x Child-resistant packaging of naproxen x Child-resistant packaging of spot remover with naptha xm Coal- and woodburning stove emissions x Dichlorobenzene x Dioxin in paper products x Exemption to child-resistant packaging of mouthwashes with ethanol x Indoor air quality x Indoor air quality—carpet emissions x Page 54 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Agency action Analytical efforts Mandatory Voluntary Information and Cost-benefit Risk assessment Ban standard standard activitya education Otherb analysis done done x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x (continued) Page 55 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Origin Initiated by the Initiated by Product Congress Petition agency Indoor air quality—formaldehyde in pressed x wood products Indoor air quality—glycol ethers x Indoor air quality—heating, ventilation, and x air-conditioning systems Indoor air quality—kerosene heaters x Indoor air quality—portable x room humidifiers Lead poisoning abatement x Methylene chloride in paint strippers x Sulfuric acid drain cleaners x Other Waterbeds x Total 115 13 34 68 Page 56 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 Agency action Analytical efforts Mandatory Voluntary Information and Cost-benefit Risk assessment Ban standard standard activitya education Otherb analysis done done x x x x x x x x x x x 2 21 55 5 32 29 24 Page 57 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix III CPSC Projects, 1990-96 a “Voluntary standard activity” includes efforts in which agency staff participated in the development of a new voluntary standard or revision of an existing voluntary standard. This participation included regularly attending meetings of a standards development group; taking an active part in discussions; and providing research or other support, information and education programs, and administrative assistance. b “Other” includes projects that were completed with no agency action, terminated, currently in process or pending, or completed with a study or report. c Two petitions were submitted for fuse-burn time. The first, submitted in 1991, was denied. The second, submitted in 1996, resulted in a change to an existing mandatory standard. d CPSC published an Advance Notice of Proposed Rulemaking (ANPR). This is the first step toward enacting a mandatory standard. e CPSC published an ANPR in 1994, which is the first step toward enacting a mandatory standard. The ANPR also solicited offers to develop or modify a voluntary standard. During fiscal year 1996, CPSC worked with an industry group on voluntary standards development. f CPSC has worked with states to develop state ATV legislation. g The baseball injuries project started in the 1980s and resulted in a special report on baseball injuries and protective equipment. The projects—baseball bases, chest protectors, faceguards on helmets, and safety baseballs—all grew out of this umbrella project and, as such, are counted separately. h A petition was also submitted in 1990. The Congress mandated this project in the Children’s Bicycle Helmet Safety Act of 1994. Project origin is counted as congressionally initiated. i The cost-benefit analysis for garage door openers applied to the labeling and record-keeping requirements only, not to the provisions of the safety standard. j The risk assessment for playground equipment examined the hazard associated with exposure to arsenic from wooden equipment. The risk assessment concluded that the risk was small. Most of CPSC’s efforts concerning playground equipment are related to injuries from falls or impacts of equipment and are not related to chemical exposure. k Petition, submitted in 1995, was withdrawn. l One petition was submitted in 1988 by the American Association of Poison Control Centers requesting CPSC to require child-resistant packaging both for household glue removers containing acetonitrile and home cold wave permanent neutralizers containing sodium bromate or potassium bromate. m Petition, submitted in 1991, was withdrawn. Page 58 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis Cost-benefit analysis can be described as an analytical technique that details the expected positive and negative effects of a given policy proposal (expected benefits and costs). To construct this framework, researchers approach a policy question needing considerable information about the potential remedy. Because cost-benefit analysis requires an inclusive approach to evaluating a proposal’s impact, the proposal itself must be well defined, and some information must be known about its impact. Once the expected benefits and costs of the proposal are thoroughly identified and delineated, the next step is to place a value on each benefit or cost. Frequently, these expected costs and benefits are expressed in numerical or monetary terms to facilitate a comparison of the aggregate costs and benefits. If all relevant factors can be translated into monetary terms, the decision rule suggested by this proceeding is to accept the proposal if its aggregate benefits exceed its aggregate costs.31 Although at first the concept behind cost-benefit analysis seems relatively straightforward, the application of cost-benefit analysis is not. The practical difficulties associated with measuring effects, quantifying results, and accounting for uncertainty (to name only a few issues) can create a gap between the way cost-benefit analysis is described in theory and the way it is implemented in practice. For this reason, experts generally agree that analysts should be comprehensive in including all important factors and be explicit in their description of the underlying data, methodology, and assumptions. This appendix describes many of the major methodological issues that often arise in cost-benefit analysis and outlines specific elements that are frequently used to evaluate cost-benefit analyses. The methodology used to conduct a cost-benefit analysis frequently varies depending on the circumstances and the context of the analysis. For this reason, there is no complete set of standards for evaluating the quality of an individual cost-benefit analysis. However, the professional literature offers some guidance for analysts, and certain specific elements are frequently used to determine whether a given analysis meets a minimum threshold of comprehensiveness and openness. These elements are necessary, but not sufficient, for a quality analysis. 31 Variations of cost-benefit analysis include cost-effectiveness analysis, which relates the costs and outcomes of alternative methods to achieve the same desired goal. For example, one cost-effectiveness analysis compared the cost per case detected for different methods of treating the same medical condition. Because only cost-benefit analysis is relevant to CPSC’s activities, related techniques are not considered here. For more information on these and other related analytical tools, see Michael F. Drummond, Greg L. Stoddart, and George W. Torrance, Methods for the Economic Evaluation of Health Care Programmes (New York: Oxford University Press, 1996). Page 59 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis From the extensive cost-benefit literature, we developed objective evaluation questions to evaluate cost-benefit analyses performed by CPSC. These evaluation questions are summarized in table IV.1. Although these evaluation questions are not a comprehensive measure of the quality of an analysis, they were designed to reflect whether an analysis is comprehensive and reported in sufficient detail. We applied these questions to each of CPSC’s analyses, but not all evaluation questions were applicable to each case. In addition, for some questions it was not always possible to determine whether CPSC’s analysis complied with the particular element reflected in the question. For these reasons, and to emphasize those areas that we considered as most critical, we reported only the evaluation results that related to key elements and that applied to the majority of CPSC’s analyses; these eight are shown in bold in table IV.1. Table IV.1: Frequently Used Elements of Evaluation of Cost-Benefit Analysis Number Evaluation question Evaluating potential courses of action 1 Was descriptive information given about well-defined alternative courses of action? 2 Was more than one alternative course of action considered in the analysis? 3 Was evidence given to support the degree to which the proposal was assumed to mitigate the problem? Considering all important factors for analysis 4 Were all important categories of potential costs and benefits included in the analysis? (For example, did the analysis include [where applicable] intangible benefits and costs, health and safety benefits, costs of compliance, upfront costs, price changes, and changes in consumer surplus?) 5 Were the effects of intangible costs and benefits detailed explicitly, even if they could not be quantified? (For example, did the analysis include [where applicable] consumer utility?) 6 Did the analysis show how the existence of intangible costs or benefits could affect the conclusions? 7a Were possible indirect effects discussed? (For example, did the analysis include [where applicable] price changes of related goods or likely changes in market concentration?) 7b If indirect effects were measured quantitatively, were the calculations and assumptions behind this measurement discussed in detail? 7c If indirect effects were measured qualitatively, did the analysis show how the existence of these factors could affect the conclusions? Considering interdependent factors 8 Were possible issues of standing identified? 9 Was an incremental (marginal) analysis of costs and benefits performed? (continued) Page 60 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis Number Evaluation question 10 Were major policy interdependencies identified? (For example, did the analysis discuss the implications if another agency also had jurisdiction over the product, or if the proposal could create a conflict with existing government policy?) Considering distribution issues 11 Did the analysis make clear the distribution of gains and losses? (For example, who would pay the costs and who would get the benefits of the proposal?) 12a Was the impact of distribution of gains and losses assessed? (For example, would the proposal be likely to have a greater adverse impact on small businesses, potentially leading to increased market concentration?) 12b Were distributional weights employed in the analysis? 12c If weights were used, was information provided about the basis for those weights? 13a Is the standard assumption that relative market prices are insensitive to the policy change likely to hold in this case—that is, are there likely to be few or no macroeconomic effects? 13b If no, was this impact discussed or considered in the analysis? Considering risk-risk factors and potential offsetting behaviors 14 Were risk-risk or offsetting behavior concerns identified and considered in the analysis? Using the willingness-to-pay approach to value risk reduction 15 Were willingness-to-pay measures used to value reductions in the risk of death or injury? 16 Is the numerical value of a statistical life used in this analysis consistent with the literature? 17 Is the numerical value of a reduction in the chance of injury consistent with the literature? Discounting future costs and benefits 18a Are costs and benefits that occur at different points discounted for differences in timing? 18b Does the discount rate or rates used include the suggested rate by the Office of Management and Budget? 18c Does the discount rate or rates used include the Treasury bill rate for the time horizon of the analysis? 18d Does the discount rate or rates used include a lower “social” discount rate? Considering data issues 19a Where applicable, were the implications of unverified or unverifiable data provided by an interested party identified and discussed? 19b Where applicable, were the implications of uncertainty surrounding a dose-response model identified and discussed? 19c Where applicable, were the implications of uncertainty surrounding crossspecies extrapolation identified and discussed? (continued) Page 61 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis Number Evaluation question 19d Where applicable, were the implications of the use of data relying on investigated or reported cases identified and discussed? 19e Where applicable, were the implications of survival bias in the underlying data identified and discussed? 19f Where applicable, were the implications of small sample sizes identified and discussed? 19g Where applicable, were the implications of other known biases in the underlying data identified and discussed? 20 If the underlying data were derived from a statistical sample, were appropriate measures of precision provided? Performing sensitivity analysis 21a Was sensitivity analysis performed on any parameter in the analysis? 21b Was sensitivity analysis performed on the value of a statistical life? 21c Was sensitivity analysis performed on the value of injury reduction? 21d Was sensitivity analysis performed on the discount rate? 21e Was sensitivity analysis performed on the precision of the underlying data? 21f Was sensitivity analysis performed on other important parameters? Note: We reported results for questions that are in bold. We viewed these measures as especially important, and they applied to the majority of CPSC analyses and a determination could be made for these questions in each case. As an organizing framework, cost-benefit analysis can help a Background decisionmaker to organize and aggregate all the relevant information in a way that can clarify the nature of the trade-offs involved in a decision. At the same time, by providing a framework to convert dissimilar effects to a common measurement, cost-benefit analysis can allow this information to be weighed and aggregated to help make a decision. A well-constructed cost-benefit analysis can highlight crucial factors, expose possible biases, and perhaps expand the openness of the decision-making process by clarifying the factors on which the decision was based—whether these factors are purely economic criteria or include other social factors. A key advantage of carefully built cost-benefit analysis is that it promotes explicit rather than implicit decision-making, even when it is impossible to monetize or even quantify all the potential effects of a given regulatory proposal. Despite the value of this analytical tool, in some situations using cost-benefit analysis as the sole basis for decision-making may be Page 62 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis inappropriate. For example, alternatives that would eliminate basic human rights or dramatically increase income inequality may be viewed as morally unacceptable. Similarly, when uncertainty about possible effects is so pervasive that attempting to identify potential costs and benefits would amount to nothing more than uninformed speculation, cost-benefit analysis probably has little to offer.32 In addition, for minor decisions it may not be necessary to employ a rigorous, time- and resource-consuming decision-making process, and a less detailed cost-benefit analysis—or none at all—may be more appropriate. Virtually all observers agree that the appropriate role for cost-benefit analysis—sole decision-making rule, input into decision-making, or not done at all—will depend on the context in which the particular decision is being considered. To some individuals, application of an abstract analytical technique such as cost-benefit analysis is especially unpalatable in certain situations. For example, the idea of applying a value to “saving life” may be distasteful, because we regard our own lives as “priceless” or of infinite value. However, although our lives may be priceless, avoiding risks often means forgoing time, convenience, enjoyment, or other opportunities—all of which do have a price. Thus, policy interventions that can affect life expectancy pose an unavoidable problem—to refuse to consider the value of potentially lifesaving interventions is to implicitly value them at zero, and to consider any potentially lifesaving activity as infinitely valuable implies that individuals would never take any action that involves risk (driving a car, for example). The literature on cost-benefit analysis makes a key distinction in this area. For ethical reasons, most practitioners consider it inappropriate to use cost-benefit analysis to evaluate alternatives that would (with certainty) affect the life expectancy of a given, known individual. But government policy does not usually involve making such decisions. Instead, policy questions typically center on actions that may bring about small changes in the statistical life expectancy of anonymous members of a large group. For example, when CPSC considered imposing a mandatory regulation on large, multiple-tube fireworks, the agency estimated that such a standard could reduce the number of individuals that die in related accidents by one anonymous 32 However, one should use caution in assuming that this situation applies. Sometimes the conclusions of a cost-benefit analysis can be quite robust even in the face of substantial uncertainty. For example, a recent analysis of the benefits and costs of reducing emissions of acrylonitrile concluded that imposing strict standards on acrylonitrile emissions fails reasonable cost-benefit tests, despite as much as a threefold factor of uncertainty in acrylonitrile emissions, additional uncertainty in the amount of exposure to acrylonitrile, and a probable but unmeasurable upward bias in the estimate of cancer risk. (See John A. Haigh, David Harrison, Jr., and Albert L. Nichols, “Benefit-Cost Analysis of Environmental Regulation: Case Studies of Hazardous Air Pollution,” ch. 1 in The Moral Dimensions of Public Policy Choice: Beyond the Market Paradigm, eds., John Martin Gilroy and Maurice Wade (Pittsburgh, Pa.: University of Pittsburgh Press, 1992), pp. 15-57. Page 63 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis consumer over a 3-year period. On a daily basis, each of us makes such trade-offs between perfect safety and other things we value, such as the convenience of driving a car or the excitement of skiing down a mountain. Such assessments, which place a value on reductions in risk, are viewed as appropriate, whereas valuing a given person’s life may be viewed as less reasonable. Similarly, a decision to surrender basic rights—such as liberty—may be unacceptable on moral grounds, and so cost-benefit analysis might not be applicable in a situation involving such rights. In addition, some individuals object to cost-benefit analysis in circumstances in which other species may be affected adversely. For example, one individual objected to a proposal that would allow increased pollution in a particular river because the proposal would adversely affect the fish and other species in the river. Cost-benefit analysis would probably be unable to address this objection because there is no accepted method to place a value on the losses suffered by the fish. Over the past 3 decades, a substantial economic literature has explored Conceptual, the conceptual, analytical, and technical issues posed by the application of Analytical, and cost-benefit analysis. From this literature, generally accepted standards of Technical Issues Arise professional practice have emerged that cover a wide range of research methods. As one experienced researcher has said, “Good studies follow When Applying procedures that are in accord with economic theory for estimating COST-BENEFIT benefits and costs, provide a clear statement of all assumptions, point out uncertainties where they exist, and suggest realistic margins of error.”33 Analysis Even when undertaken by careful and competent researchers, cost-benefit analysis can sometimes be difficult to interpret, especially when uncertainty is substantial and information is incomplete. Conceptual Issues in Some of the issues underlying the application of cost-benefit analysis are Cost-Benefit Analysis conceptual, and the researcher may make a different judgment in different circumstances. With respect to these issues, being complete and explicit is important—the consensus in the literature is that while there may be no single method for dealing with these issues that is universally appropriate, the researcher must be clear and direct in detailing how the issue was addressed in the context of the analysis. 33 See Eban Goodstein, “Benefit-Cost Analysis at the EPA,” The Journal of Socio-Economics, 24 (2) (1995), pp. 375-89. Page 64 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis Intangible or Unquantifiable Ideally, a cost-benefit analysis involves translating each impact into a Factors Can Pose Dilemmas for common measurement (such as dollars) for comparison. However, some Researchers effects may be difficult or impossible to measure or quantify. For example, a researcher evaluating an alternative outpatient mental health treatment program realized that the greater independence afforded by the outpatient program could create increased anxiety—or, alternatively, higher self-esteem—for some participants. However, these intangible effects would be very difficult to measure and even more difficult to quantify.34 Some individuals have criticized certain individual practitioners of cost-benefit analysis for ignoring or de-emphasizing aspects of proposed changes that cannot be easily quantified. Although it is necessary for such effects to be described and emphasized appropriately, the existence of such intangibles does not necessarily limit the value of the analysis. For example, in the discussion of mental health programs, the researchers found that patients in the experimental program experienced higher satisfaction and reported having a greater number of social relationships. These qualitative benefits were important to the analysis even though they could not be valued in dollars. In addition, if it can be shown explicitly that the value of the intangibles would be unlikely to change the conclusion, then cost-benefit analysis has played a valuable role by considering intangible effects explicitly, even if it is not possible to consider them quantitatively. (We addressed this issue in questions 5 and 6 of table IV.1.) Measuring Indirect and Researchers must also decide whether or not to include indirect or Secondary Costs and Benefits secondary effects that may result from the proposal. For example, a Can Be Difficult change in tax rules may not only have an initial, direct effect on individuals’ income but may also create a secondary ripple (or “multiplier”) effect on the economy as a whole. Similarly, a medical treatment that prolongs life can be expected to have a secondary effect on health care costs as individuals live longer. A researcher who wants to include such secondary impacts will frequently find a measurement challenge, because determining the magnitude (or perhaps even the existence) of many secondary effects involves answering a “what if?” type of question, and frequently without relevant historical experience. For example, individuals who receive improved treatment for hypertension may contract more prolonged or costly illnesses in the future.35 Any attempt to measure the impact of these indirect future costs 34 See Burton A. Weisbrod, “Benefit-Cost Analysis of a Controlled Experiment: Treating the Mentally Ill,” Journal of Human Resources, 16 (4) (1981), pp. 523-48. 35 For an example of such an analysis, see W.B. Stason and M.C. Weinstein, “Allocation of Resources to Manage Hypertension,” The New England Journal of Medicine, 296 (13) (1977), pp. 732-39. Page 65 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis would require the researcher to predict the future health status and health care costs of the patient group in the event that the treatment had or had not been available. In addition, the existence of some secondary effects is dependent on one or more key assumptions. For example, an economic multiplier effect is generally thought to take place only in an economy operating below its productive capacity. A researcher who wants to include such an effect must thus determine whether or not this assumption holds, and then decide how large an effect should be included. Secondary impacts may be considered part of the cost-benefit analysis or extraneous to it. The proper choice in each circumstance probably depends on the likelihood that the secondary effects in fact exist, their probable importance, and the ability to measure them. However, like intangible effects, potential secondary or indirect effects should be detailed even if they are not included quantitatively. Furthermore, should such effects be included, the assumptions underlying their existence and measurement must be revealed explicitly to allow the reader to make an informed judgment. (We addressed this issue in question 7 of table IV.1.) Issues of “standing” May In some circumstances, it may be difficult to define the point of view from Require Clarification of the which the cost-benefit analysis is calculated—that is, who has “standing,” Scope of the Analysis or whose costs and benefits should be included in the analysis. For most cost-benefit calculations involving government policy, the analysis is appropriately done at the level of “society” (rather than, for example, considering only the impact on individuals in a particular state or locality), so that a wide range of implications is considered. However, in some circumstances, it is important to make the point of view explicit. For example, when policy effects may cross national boundaries, “society” may be best defined on a broader basis. If the cost-benefit analysis is considering policies to reduce acid rain, for example, considering the effects of acid rain and pollution reduction on the Canadians who share U.S. weather patterns may be important. Similarly, analysts may wish to consider the impact of product regulation on foreign producers as well as on U.S. consumers. Even when the unit of analysis is clear, occasionally some members of that unit are not afforded standing under cost-benefit analysis. For example, suppose that a particular policy proposal is likely to result in a decrease in property crime. If a thief steals a television, and both the victim and the thief are given standing, then the gain to the thief could offset the loss to the victim in a cost-benefit accounting. “Society,” however, is clearly worse off. Unless we try to measure the psychological and sociological Page 66 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis costs of crime, it may make more sense not to afford the thief standing in the cost-benefit analysis. Such issues arise infrequently but should be made clear if they could affect the interpretation of the analysis. (We addressed this issue in question 8 of table IV.1.) Taking the Status Quo as Some cost-benefit analyses have been criticized for excluding relevant Baseline, Cost-Benefit Analysis factors on what appears to be an ad hoc basis. Without including such Should Account for Relevant factors, or providing an explicit justification for excluding them, a Factors and Constraints researcher limits the value of the analysis. However, while it is important to consider a variety of alternative actions, it is equally crucial to adopt a realistic view of the possible alternatives. For example, political and agency realities often place meaningful constraints on options available for consideration. If these constraints are not incorporated into the analysis, the results could be “pie in the sky” recommendations that would be of little use of the decisionmaker. (We addressed these issues in questions 1, 2, 4, and 10 of table IV.1.) Similarly, it is important that cost-benefit analysis be conducted in a way that evaluates the change in aggregate costs and benefits, with current conditions serving as a baseline. For the properties of economic efficiency to hold, cost-benefit analysis must take what economists call a marginal or incremental approach—that is, it must consider only the changes that would result from the proposed intervention. For example, a proposal to renovate a museum should be evaluated, for cost-benefit purposes, on the basis of the incremental cost of the renovation, not on the basis of the original cost of constructing the entire museum. (We addressed this issue in question 9 of table IV.1.) Issues of Distribution May Be By definition, cost-benefit analysis is a method for considering the Considered Within or Outside aggregate effects of a given proposal. If the aggregate expected benefits of the Cost-Benefit Context the proposal exceed the aggregate expected costs, the proposal is said to pass the cost-benefit test. This does not necessarily mean that adopting the proposal would improve the condition of each individual; instead, it implies only that when the expected gains and losses to all individuals are added up, the total expected gains exceed the total expected losses.36 36 Economists make a distinction between events that lead to a “Pareto improvement” and those that lead only to a “potential Pareto improvement.” If a change leads to a Pareto improvement, then as a result of the change no individual will be worse off, and at least one individual will be better off. However, if a change leads to a “potential Pareto improvement,” then the aggregate gains exceed the aggregate losses, but some individuals will gain and others will lose. In this situation, it would be theoretically possible for the “winners” to compensate the “losers” and leave no individual worse off than before if redistribution itself is costless. Page 67 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis By adding up individual gains and losses to determine the effect on society as a whole, cost-benefit analysis implicitly assumes that each individual’s gains or losses should be valued equally with any other individual’s gains or losses. As a result, cost-benefit analysis is “neutral” with respect to the distribution of gains and losses. To put it another way, the “fairness” of how gains and losses are distributed is generally not included in the calculations underlying a cost-benefit analysis. Sometimes a decisionmaker might want to address such issues of fairness. For example, if a proposal would involve redistributing income, the loss of one dollar to a wealthy person may be viewed as less consequential than the gain of one dollar to a poor person.37 In order to address such an issue, the researcher or decisionmaker can consider issues of distribution inside or outside the cost-benefit context. One method of examining such issues might be to analyze the distributional impact of a proposal separately, and consider fairness issues—along with the cost-benefit calculation—as another factor in the decision-making process. Under these circumstances, if the proposal is viewed as having negative distributional consequences, and it would be costly or difficult to redistribute the gains and losses, then the proposal might be rejected even though it would otherwise meet a cost-benefit test. Occasionally a researcher who wants to consider distributional issues explicitly chooses to incorporate distributional consequences into the mathematics of the cost-benefit calculation. Instead of simply adding up the costs and benefits accruing to each individual, the researcher uses a mathematical formula that applies different weights to different individuals. These weights could be based on a number of factors, depending on the characteristics of the “fairness” issues being addressed. For example, if the proposal would affect income directly, weights could be based on income (with greater weight being applied to gains in income experienced by low-income individuals). Weights could also be based on other circumstances; for example, greater weight could be placed on vulnerable populations or on future generations. Similarly, if a change was proposed in a given program, greater weight could be applied to the 37 There is some limited ground in utilitarian economic theory for this assumption. Economic theory states that individuals will generally, at some point, experience “diminishing marginal utility of income”—that is, receiving one additional dollar of income brings less additional satisfaction (or “utility”) as the individual becomes wealthier. However, the amount of satisfaction deriving from any good or service, including income, cannot be compared across individuals. As a result, while economic theory suggests that it is plausible that at some point giving one dollar to a rich person produces less total “utility” than giving one dollar to a poor person, economic theory also tells us that it is generally not possible to measure the circumstances under which this assumption might (or might not) be correct. Page 68 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis smaller number of program participants (who would be more significantly affected) than the rest of society (who would be affected only indirectly). A weighting scheme would be most helpful to decisionmakers and outside observers, however, if the formula was detailed explicitly and the analysis was accompanied by a sensitivity analysis, so the effect of the distributional weights was clear to the reader. (We addressed these issues in questions 11 and 12 of table IV.1.) Implementation Issues in While a number of conceptual issues arise in cost-benefit analysis for Cost-Benefit Analysis which the appropriate answer depends on circumstances, there are also a number of methodological or implementation issues about which there is widespread agreement. For example, years of debate on the appropriate valuation method for risk reduction have largely been resolved in the favor of “willingness to pay” measures, and the importance of sensitivity analysis is generally recognized. While the choice of discount rate has long been a subject of controversy, with some practitioners arguing for the use of market rates and others advocating a lower “social” discount rate, the literature generally recognizes the value of multiple rates. Finally, there is virtually universal agreement on the importance of reliable data and careful risk measurement to cost-benefit analysis. Costs and Benefits Should Be Frequently, the various consequences of the proposal under discussion Adjusted for Differences in will differ in when they occur. For example, changing the labeling Timing requirements on bags of charcoal could reduce carbon monoxide deaths years later and also result in an immediate, one-time increase in industry costs. As individuals and as a society, we generally prefer to have dollars or resources now than at some time in the future because we can benefit from them in the interim. In addition, if we acquire $1 next year instead of today, we give up the opportunity to invest that dollar and earn interest on our investment. As a result, it is generally agreed that future dollar cost and benefit streams should be reduced or “discounted” to reflect differences in timing. The rate at which this adjustment is made is usually done on a case-by-case basis. Many analysts choose to use the market rate of interest—for example, the rate payable on government bonds with a time horizon comparable to the analysis. Some researchers have argued that the discount rate should be set somewhat higher because, presumably, economic growth will leave future generations better off. Others have advocated using a “social” discount rate, which is generally lower than the market rate, on the grounds that society’s interest in the welfare of future Page 69 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis generations implies that the discount rate for all of society should be lower than the rates chosen by individuals. Experts on cost-benefit analysis generally encourage researchers to use multiple rates—both to assess the sensitivity of the results to the chosen discount rate and to facilitate comparison across studies. (We addressed issues of discounting in question 18 of table IV.1.) Comparability With Other Ideally, proponents of extensive cost-benefit analysis would like to be able Analyses Could Influence to order and prioritize regulatory interventions across agencies. A Selection of Parameters potential mechanism for this type of coordination is provided in Executive Order 12866, which requires agencies to submit to the Office of Management and Budget (OMB) cost-benefit analyses for regulations with an estimated impact on the economy of $100 million or more.38 OMB has suggested or prescribed certain rules (such as a presentation format and/or a specific discount rate) to facilitate comparability across agencies and to help promote quality analyses.39 While a common set of assumptions or parameters may facilitate comparability by a central government authority, so that priorities may be set across as well as within agencies, some of these assumptions may fit the circumstances of the given analyses less well. For example, a given discount rate may be more appropriate for one project (for which the benefits and costs are spread out over a long period) than another (with costs and benefits spread over a shorter period). Therefore, there may be a tension between making an analysis comparable to others and customizing it to fit a unique situation. (We addressed these issues in questions 10 and 18 of table IV.1.) Potential Future Price Changes Typically, cost-benefit analysis assumes that the intervention considered Can Add Uncertainty to has only a small or negligible effect on relative prices throughout the Analysis economy. Generally, this assumption makes sense; however, for major changes (such as a big change in tax law) the assumption is clearly inappropriate. In these cases, it is much harder to put a value on the potential impact of the proposed change. For example, a large cut in the capital gains tax could affect a wide range of investment behavior, and may even affect how much some individuals choose to work and how much they are paid. A researcher trying to place a value on the potential consequences of a capital gains tax cut might introduce an error by valuing changes in productivity using the wage rates that prevailed before the cut, because, in reality, once the new policy has been implemented, wages rates might change. In such a situation, it is especially important for 38 CPSC, as an independent agency, is not required to follow these procedures. 39 The evaluation factors we examined parallel OMB’s suggestions; in some areas, we have operationalized these concepts to create more specific and objective evaluation questions. Page 70 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis the researcher to point out the potential mismeasurement, and (if feasible) try to model its effect. (We addressed these issues in question 13 of table IV.1.) Researchers Recognize the Every cost-benefit analysis will include some level of uncertainty or Importance of Sensitivity imprecision, or reflect a methodological choice that not everyone will Analysis to Inform the Reader necessarily agree with. Careful analysts will identify critical sources of uncertainty or controversy, and revise or test the analysis quantitatively or qualitatively to identify how or whether these areas affect the conclusions reached. If large variations in measurement or assumption do not alter the conclusions, then the researcher and the decisionmaker can have greater confidence in the original results. However, if the conclusions of the analysis can change depending on methodological choices or variable measurement, then the researcher may want to try to improve the measurement of the original variables. If this is not possible, then the analysis may be of more limited value—an implication the decisionmaker would need to know. (We addressed these issues in question 21 of table IV.1.) Reliable Underlying Data Despite the importance of sensitivity analysis, even the most careful and Crucial to Reliable Cost-Benefit elaborate sensitivity analysis often cannot sufficiently compensate for Analysis poor underlying data. Obviously, if key variables are defined poorly or measured imprecisely, the quality of the cost-benefit analysis will suffer. Because many variables used in research are measured from surveys or samples, sampling error may be unavoidable. The researcher can use a sensitivity analysis to test the potential for sampling error to affect the conclusions; however, if the survey instrument itself is flawed, the results may be unreliable even beyond the degree indicated by sampling error. Similarly, researchers may face a difficult problem when key data (such as the cost of compliance) are provided for the analysis by an interested party (such as the industry). If the interested party provides the data, that party may have an incentive to provide biased, inaccurate, or misleading information. In such cases, it is important to try to verify the data to the extent possible; if not, the researcher needs to at least note the source of the underlying information and allow the reader to make an informed judgment about the reliability of the final analysis. (We addressed potential data issues in questions 19 and 20 of table IV.1.) Measuring and Valuing Frequently, the benefits of a particular proposal involve a reduction in risk Risk Reduction Under of injury or death. In these situations, the quality of the cost-benefit Cost-Benefit Analysis analysis will also depend substantially on the estimates of the proposal’s Page 71 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis impact on these risks. However, it is often difficult to accurately measure and value risk reduction. A number of issues arise, including the effect of the proposal on the level of risk (especially when individual behavior is a factor in determining the size of the risk) and measuring individuals’ willingness to take risks in exchange for rewards. Considerable Uncertainty Often Measuring the effect of a proposal on the level of risk can be especially Surrounds Risk Measurement difficult. Generally, it is not realistic to assume that any regulatory and Risk Assessment intervention will reduce the risk level to zero—removing an environmental carcinogen, for example, will probably reduce the number of cancer deaths, but is unlikely to eliminate cancer entirely. Considerable uncertainty may be inevitable when extrapolating a dose-response model. Sometimes large doses of a potentially dangerous substance, given over a short period of time, are used to predict the results of long-term exposure. When the model moves across species—for example, predicting cases of cancer in humans based on experiments with laboratory animals—another source of uncertainty is introduced. Measurement problems in the underlying data can make it hard to predict future risk from epidemiological data as well. For example, when longitudinal data are used, there may arise a “survival bias” in that the analysis could be biased if it excludes individuals who die or otherwise move out of the data set. Similarly, when some cases go unreported, the data may understate the size and/or overstate the severity of the hazard. For rare hazards, limited variation in the epidemiological data can make measurement and prediction of risk more difficult. In addition, some data may be known to produce biased estimates (because of exclusions or potential double-counting, for example). Data problems such as these (and the statistical or analytical methods used to deal with them) should be pointed out to the reader to increase the opportunity for informed judgment. (We addressed these issues in questions 3 and 19 of table IV.1.) Risk-Risk Trade-Offs and Sometimes actions taken to reduce one risk can have the unintended Individuals’ Behavioral effect of increasing that or another risk. For example, unforeseen Responses to Policy Changes consequences arose in the 1970s when CPSC issued regulations requiring Can Affect the Level of Risk children’s sleepwear to meet flammability standards. Manufacturers used a chemical called Tris to meet these standards, but later it was discovered that Tris posed a cancer risk. Changes in individual behavior can also create uncertainty in predicting the level of risk because valuations of costs and benefits of a proposed action are often based on historical behavior. For example, valuations of a proposed new highway may be based on the number of individuals currently traveling the route, their Page 72 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis commuting times, and other factors. However, once the new highway is built, some people may make extra trips in the area (shopping in different stores, for example) than they did before. Some people who had driven at off-hours to avoid congestion on the old road may now travel at peak hours on the new one. Similarly, changes in public policy may cause individuals or firms to change the technology they use, the amount of time they spend at leisure or at work, or the amount they invest in innovation. A special type of behavioral change—often referred to as “offsetting behavior”—occurs when individuals change the amount of precautions they take in response to a change in policy. For example, having an air bag in the car or being required to wear a motorcycle helmet might make some drivers feel safer, so they exercise less caution on the road. Sometimes this offsetting behavior can result in an increase in the risk the intervention was designed to mitigate. For example, because child-resistant medicine bottles can be difficult to open, a grandparent may leave the cap off the bottle, creating an even greater risk than would be posed with a non-child-resistent cap. If such changes in consumer behavior are foreseeable, the analysis will be improved if the researcher points out such possibilities, and it will be even more useful if reasonable attempts can be made to measure the impact. (We addressed these factors in question 14 of table IV.1.) Willingness-To-Pay Measures Measuring the benefit of risk reduction requires placing a value on Viewed as Appropriate for avoiding death and/or injury. Several approaches are available for this Valuing Risk but Require task, but two have been used in cost-benefit analysis: (1) the “human Careful Estimation capital” approach, in which death or injury is valued at the market value of the lost production it causes plus the medical costs expended and (2) the “willingness-to-pay” approach, which attempts to measure directly individuals’ willingness to pay for reducing the risk of death or injury.40 In early cost-benefit analysis, the human capital approach was used, largely because lost wages and medical costs were relatively easy to measure. However, this approach is not preferred today because of a number of shortcomings. First, if this approach were taken literally to apply to individuals, then persons who do not produce output in the marketplace—such as the elderly or homemakers—are not valued—an assumption that is clearly ethically and economically inappropriate. In addition, the human capital approach is unable to take into account costs to the individual of death or injury such as pain and suffering. Also, it is difficult to apply an “average” value based on human capital calculations 40 This terminology has been borrowed from W.K. Viscusi in “The Valuation of Risks to Life and Health,” in Benefits Assessment: The State of the Art, eds., Judith D. Bentkover, Vincent T. Covello, and Jeryl Mumpower (Dordrecht, Holland: Reidel Publishing Company, 1986), pp. 193-210. Page 73 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis to a statistically anonymous member of a large group. Finally, we are all more valuable than the sum of what we produce, and these amounts could not be included in a human-capital-based formulation. Thus, human capital measurements are generally viewed as a lower boundary on the value of avoiding death and injury, and are less preferred than the more recent willingness-to-pay measures. In part because of dissatisfaction with human capital measures as used in cost-benefit analysis, and facilitated by newly available large micro-level data sets, economists measure the benefits of reducing death or injury by calculating the consumer’s willingness to pay for small reductions in the probability of injury or death. These calculations have been done several ways. Some approaches attempt to glean willingness-to-pay measures from observed behavior in the marketplace. For example, one approach examines pay differentials for jobs with different risks. Another approach looks at the payoff to consumers who purchase safety devices, such as smoke detectors and air bags in automobiles. These measures have the advantage of being based on actual observed behavior in the marketplace rather than on an artificial experimental situation. However, the validity of such measures is based on the questionable assumption that workers and consumers are sufficiently knowledgeable about the risks they face and potential of different occupations or safety devices to alleviate those risks. For many purposes, practitioners of cost-benefit analysis can select an appropriate value from the range of research already done in this field without performing the actual analysis themselves. However, for other analyses, especially those involving unique risk-taking situations, it may be wiser to gather new data to construct an estimate that is based on circumstances as close as possible to those being studied. Another common method for valuing risks is known as contingent valuation, in which such values are elicited by observing responses or behavior on a survey or in a controlled experiment. For example, researchers surveyed individual shoppers on how much of a premium they would be willing to pay for pesticide-free grapefruit.41 These methods have the advantage of being able to provide information on areas that cannot be addressed with market data. However, this characteristic could also be a weakness—the very artificiality of the situation could make the consumer make a less deliberate choice or could limit the usefulness of applying this measure to other situations. This method does entail some technical requirements—for example, it may be useful to perform statistical tests on 41 Jean C. Buzzby, Richard C. Ready, and Jerry R. Skees, “Contingent Valuation in Food Policy Analysis: A Case Study of a Pesticide-Residue Risk Reduction,” Journal of Agricultural and Applied Economics, 27 (2) (Dec. 1995), pp. 613-25. Page 74 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix IV A Review of the Literature on COST-BENEFIT Analysis the distributional assumptions when constructing contingent valuation measures. (We addressed these issues in questions 15 through 17 of table IV.1.) Page 75 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC In order to target its resources and analyze the costs and benefits of projects or potential projects, CPSC must obtain data on injuries and deaths related to particular product hazards. CPSC relies on a patchwork of independent data systems to address this need. However, not only does each of these data sources have its own internal limitations, but together CPSC’s data sources present an incomplete—and potentially distorted—picture of consumer-product-related injuries and deaths. The implications of this lack of data range from reducing CPSC’s ability to apply regulatory project selection criteria to limiting the agency’s ability to estimate the impact of its regulatory actions. CPSC Relies Heavily on Hospital Emergency Room Reports to Obtain Estimates of Injuries NEISS System CPSC obtains most of its injury information from its National Electronic Injury Surveillance System (NEISS), which gathers information from hospital emergency room records. NEISS provides national estimates about the number and severity of emergency-room-treated injuries associated with, although not necessarily caused by, consumer products in the United States. To accomplish this, a stratified probability sample of hospitals is drawn that is representative of all hospitals with emergency departments in the United States and its territories.42 CPSC constructs national estimates of the number of injuries associated with individual consumer products on the basis of reports from this sample of 101 hospitals. NEISS data result from information abstracted from hospital emergency room records by coders trained by CPSC staff. The data are coded and entered, at each site, into a personal computer programmed for this purpose. The software checks the data entries for consistency. The data collections are linked nightly to a CPSC permanent central database. Data about product-related injuries are available to CPSC staff within 72 hours after the accident for most of the injuries reported. These daily inputs are reviewed by CPSC staff for quality and for identifying possible emerging 42 The hospitals are sampled from a universe in which each hospital has at least six beds and provides 24-hour emergency service. Page 76 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC hazards. The timeliness of the data also allows staff to observe seasonal or episodic variations. For example, during a 30-day period surrounding the Fourth of July in 1990, CPSC gathered extra data through NEISS for a special study on injuries involving fireworks. The unit of analysis for NEISS is the injured person. Other key characteristics coded include the date of treatment, age and gender of patient, injury diagnosis, body part affected, disposition of case, product involved, and accident location. In addition, important details about the injury and the injured person are provided in NEISS. For example, the address and phone number of the injured person is included, permitting follow-up investigations about the nature and cause of the injury. There are about 900 product codes, ranging from abrasive cleaners to youth chairs, which the coders use to specify the product involved.43 Consumer products are coded to allow for a great deal of specificity in the estimates. For example, a “hand saw” would be differentiated from a “portable circular power saw,” and a “bicycle-mounted baby carrier” would be specified differently from a “backpack baby carrier.” However, NEISS’ coding system that describes the diagnosis of the primary injury and the body part injured is not overly specific, and because it is unique it cannot be directly compared with similar data from other databases. Despite the extent of valuable information provided by NEISS, this system also has significant limitations. One important consideration relates to the nature and size of the NEISS sample design. NEISS, throughout its history, has been designed so that only national—not state, local, or regional—estimates can be made. Thus, NEISS cannot detect interregional or interstate differences, and may also be limited in its ability to identify emerging product hazard patterns that are focused in specific states or regions. A limitation in the NEISS data for which CPSC has received criticism is the lack of information that would assist in assessing causality—that is, information that would establish (or provide a starting point to establishing) whether the product in question caused the accident or merely was involved in the accident. NEISS contains neither “N” codes—which describe the nature of the injury—nor “E” codes—which help explain how the injury happened. E codes briefly describe the circumstances of the accident that produced the injury. For example, E codes could help distinguish among falls that occurred on stairs, from a 43 Customarily excluded from NEISS are injury data involving automobiles and motorcycles, trains, boats, planes, firearms, food, illegal drugs, pesticides, cosmetics, medical devices, assaults and suicide attempts, and occupational injuries. Page 77 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC ladder, or off a roof. NEISS’ short narrative section sometimes contains this type of information, but NEISS does not generally provide much information on the circumstances surrounding the product’s involvement. Other Sources of Data on CPSC augments the NEISS injury reports with other sources of anecdotal Injuries information. For example, CPSC maintains an Injury or Potential Injury Incident (IPII) database of reports the agency receives about injuries or incidents involving consumer products. These reports come from a variety of sources, including news clips; consumer complaints, including calls to CPSC’s hot line; and public reports of product liability suits. Although the IPII can provide only anecdotal information, this database sometimes contains more detailed information than can be found in NEISS. For example, IPII records can contain specific information about the product involved—such as the manufacturer and date of purchase—that are generally not found in NEISS. The IPII also serves as a source for cases to investigate. This is particularly important for hazards for which NEISS provides relatively few cases. CPSC has also purchased data on poisonings in the form of the American Association of Poison Control Centers (AAPCC) database. This database is composed of reports from approximately 65 poison control centers throughout the country.44 Reports from the AAPCC database contain such information as the number of phone calls received by participating centers concerning possible ingestion of a product and the number of individuals who reported experiencing symptoms related to the ingestion. But these reports contain little other information—for example, they do not show how much of the poison was ingested. Although NEISS provides some information on fire-related injuries, CPSC obtains additional data on fires from the National Fire Protection Association (NFPA) and the U.S. Fire Administration. NFPA, a private organization, conducts an annual survey of fire departments that is designed to make statistically valid national estimates of the total number of fires experienced nationally each year. However, the NFPA survey does not collect much detailed information about the characteristics of individual fires. To augment the NFPA information, CPSC relies on the more detailed information provided in the National Fire Reporting System 44 The number of poison control centers reporting cases to the AAPCC database varied over the years, ranging from 47 centers in 1984 to 67 in 1996. Page 78 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC (NFIRS), which is compiled by the U.S. Fire Administration.45 NFIRS data can provide information on the ages of the victim or victims, and the type of dwelling (apartment versus single-family home, for example). However, these data are available only with a long lag; a CPSC official we interviewed in October 1996 told us that the most recent data he had available at that time was from 1993. CPSC obtains the majority of its information on fatalities from purchasing CPSC Obtains Most of death certificates from the states and culling information from them to Its Fatality determine which deaths were related to consumer products. (Like the Information From NEISS emergency room reports, the death certificates establish only involvement—not causality—of a consumer product.) CPSC does not Selected Death purchase the complete annual set of death certificates from each state; Certificates instead, the agency buys death certificates according to selected E codes.46 CPSC purchases about 8,000 death certificates annually; of these, approximately 50 percent will be related to a consumer product. Over the years, in response to declining budgets, CPSC has reduced even further the number of E codes for which it purchases certificates. Death certificates include the date, place, cause of death, age, gender, race, and residence of the deceased. Although death certificates cannot indicate whether the consumer product was “at fault” in a death, they do provide some information on the underlying circumstances. For example, the E code would specify that a person was killed in an accident caused by electric current. Of course, the quality of the data depends on the care with which causation is determined and reported. It is likely that this differs from locality to locality as well as among individual doctors. However, there are some objective indications that the quality of reporting causes of death has been improving; for example, the proportion of cases that are categorized under 45 Although NFIRS is a large dataset, it does not capture data uniformly across states. CPSC generally assumes that NFIRS data accurately depict the distribution of characteristics across fires, and then applies the NFIRS proportions to the NFPA data to approximate these characteristics on a national level. 46 According to agency officials, the major categories of E codes the agency does not purchase involve falls (which infrequently involve consumer products) and fires (for which the agency relies on other sources). This means that fall-related deaths associated with consumer products—such as falls from playground equipment—will be undercounted. In 1995, CPSC conducted a pilot in which it purchased a complete set of E codes from one state to determine whether it should change its mix of E codes. CPSC decided that the set it was buying was still the best choice given the budget for death certificate data. Page 79 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC ill-defined conditions has been falling.47 Although death certificates do not constitute a sample of known probability or a complete count of fatalities, and thus statistically reliable estimates cannot be made, the geographic information contained in death certificates may help to identify state and regional patterns as well as those that are national in scope. However, death certificate data also have substantial limitations. First, there is an extensive lag—usually 2 years—before data become available. Therefore, death certificates are not very useful in timely identifying emerging hazards. Furthermore, a number of factors contribute to obtaining rather sketchy causal information. For example, details are quite limited, which inhibit determining the degree to which a consumer product was involved in the death, let alone whether the product was defective or hazardous. Certificates are frequently completed without the benefit of autopsy information to establish the precise cause of death, either because an autopsy was not performed or because the certificate was filled out before the autopsy took place. There is substantial variation in coding practices and the level of detail available from state to state.48 The information available about the injured individual is very limited, and the number of cases for most categories is very small; such data therefore limit the kinds and amount of analyses CPSC can perform and the conclusions it can draw. CPSC augments its death certificate data with other sources where possible. For example, CPSC has instituted a Medical Examiners’ and Coroners’ Alert Project (MECAP), to provide more timely fatality information that lends itself to follow-up investigations. CPSC has engaged some 100 coroners and medical examiners from across the country to report potential product-related hazards. This project produces about 2,000 reports annually, and CPSC staff credit the coroners’ reports with alerting them to a suffocation hazard concerning infant cushions, which eventually led CPSC to recall existing products and ban future production. In addition to the MECAP data, CPSC also records some reports of fatalities in the IPII, NEISS, and NFIRS databases. 47 See National Center for Health Statistics, Advance Report of Final Mortality Statistics for 1994, 45 (3) (Sept. 30, 1996), p. 74. 48 For more information on this variation, see New England Journal of Medicine, Vol. 313 (1985), pp. 1285-86, and U.S. Department of Health and Human Services, Morbidity and Mortality Weekly Report, Vol. 37 (1988), pp. 191-4. Page 80 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC In order to effectively target resources, identify hazard patterns, and CPSC Investigates determine the appropriate remedy for particular product hazards, CPSC Selected Incidents to needs detailed analyses of the causes of reported incidents. CPSC’s data Obtain Detailed sets generally do not provide such information. The NEISS data do not include E codes (standardized, if brief and incomplete, descriptions of Information About the incident circumstances), nor do they include detailed information about Causes of Deaths and how the incident happened. Some information may be provided in the short, free-text comment area of the NEISS report, but generally few such Injuries details are recorded. Death certificates do include E codes, but product involvement is often difficult to ascertain. As a result, CPSC staff perform follow-up investigations on selected cases to develop additional information about each incident. Some of these investigations are conducted entirely by telephone, while others are conducted at the accident site. These investigations may include detailed interviews with victims and/or witnesses, police or fire reports, photographs of the product and/or the accident site, laboratory testing of the product involved, or re-creations of the incidents. For example, in 1996 CPSC staff investigated an incident in which a baby’s leg was scratched as it was caught between the slats of her crib. As part of the on-site investigation, the CPSC investigator interviewed the child’s mother, examined the child, examined and photographed the crib, and interviewed staff at the store where the crib was purchased. The CPSC staff we interviewed told us that investigations, particularly on-site investigations, were an important source of information on established projects. The additional detail these investigations gathered helps determine causality and identify hazard patterns, leading analysts to the appropriate remedies. For example, investigations revealed that very few bicycle accidents were related to mechanical problems, and as a result, CPSC staff decided not to recommend any changes to existing bicycle standards. In addition, investigations may provide key evidence to help identify and correct compliance problems. For example, the investigator who reviewed the crib incident found that the crib in question appeared to violate several mandatory safety standards. Page 81 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC In order to produce a numerical risk assessment, CPSC must have some CPSC Uses Modeling information on the extent to which individuals are exposed to a particular Techniques and product hazard. Exposure information can take many different forms, and Special Surveys to the best measure of exposure will depend on the characteristics of the particular product hazard. For example, one measure of exposure might Estimate Exposure be the number of products in use, while another measure might be the Information number of hours a person spends using the product, and another measure might take into account the intensity with which a product is used. If a product is for one-time use only and is usually used soon after it is sold (such as fireworks), number of products sold might be a reasonable proxy for exposure. However, when a product is more durable, and used for a longer period, and often “handed down” to another user, such as a baby high chair, it would probably be more reasonable to base estimates of exposure on the number of products in use. In cases where the potential hazard is especially ubiquitous (like air pollution, for example), population measures (the number living near the source or the number of children under 5, for example) may be reasonable. For some hazards, it may be important to account for the intensity of use. For example, the probability of developing cancer from exposure to a wood stove may depend on how often the stove is used, how large a space it heats, and how many times the stove door is opened to add wood. Similarly, bicycles could be used very frequently and very intensely (every day in urban traffic) or infrequently and not intensely (once a season on the bike path in the park); thus, a good exposure measure would take such factors into account. CPSC does not conduct a formal, numerical risk assessment for each project it undertakes. Of the 115 CPSC projects we reviewed, only 24 included a numerical assessment of risk. CPSC most frequently relied on estimates of products in use to provide the exposure information for its risk assessments. In 65 percent of the cases where an epidemiological risk assessment was performed, CPSC based its exposure measure on an estimate of the number of products in use.49 In an additional 5 percent, CPSC obtained information on the actual number of products in use. In 30 percent of cases, CPSC used a population-based measure, and in 10 percent of cases, CPSC used a sales measure.50 We did not evaluate whether the exposure measure CPSC chose was appropriate for each case. CPSC conducted special surveys on bicycles and on cigarette lighters and matches, for instance, to develop exposure information. The survey 49 About half of these estimates of products in use were derived from modeling and about half from special surveys. 50 Percentages do not total 100 because in two cases CPSC used more than one measure of exposure. Page 82 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC questions were designed to obtain information on the number of products in the home, the intensity of use, the characteristics of the households using the product, and the usual patterns of use. For example, the bicycle survey included questions on the number of hours spent biking, the ages, education, and income of household members, whether riding was done most often on streets, sidewalks, or bike paths, and whether riders used helmets. Where special surveys are not practical—because of time, resource, or other limitations—CPSC sometimes uses mathematical modeling techniques to estimate the number of products in use. These models can take sales information and information on the life of the product to estimate the number of units in use. For example, CPSC used such a model to estimate how many of the portable heaters made before a 1991 revision to a voluntary standard were still in use. CPSC may also receive information about potential product hazards CPSC Obtains Some through industry reporting requirements. Although this information is used Information Under mostly for identifying and addressing compliance problems, it may also Reporting help identify new hazards. Companies are legally obligated to report to CPSC information they receive that indicates a consumer product they Requirements for distribute is potentially hazardous. Under section 15 of the Consumer Manufacturers and Product Safety Act, manufacturers (including importers), distributors, and retailers of consumer products must notify CPSC if they obtain information Other Businesses that a product (1) fails to comply with a consumer product safety regulation or a voluntary consumer product safety standard, (2) contains a defect that could create a substantial product hazard, or (3) creates an unreasonable risk of serious injury or death. In addition to these reporting requirements, manufacturers and importers of a consumer product must report to CPSC if (1) a particular model of a consumer product is the subject of at least three civil actions that have been filed in federal or state court, (2) each suit alleges the involvement of that model in death or grievous bodily injury, and (3) within a specified 2-year period at least three of the actions resulted in a final settlement involving the manufacturer or importer or in a judgment for the plaintiff. Manufacturers must file a report within 30 days after the settlement or judgment in the third such civil action. CPSC may receive requests for information that firms have reported to it under section 15 requirements. The law limits CPSC’s disclosure of any Page 83 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix V Death, Injury, and Incident Data Sources Used by CPSC information identifying a manufacturer and further limits CPSC’s release of information that firms have provided under these requirements. Reports on civil lawsuits may not be publicly disclosed by CPSC or subpoenaed or otherwise obtained from CPSC through discovery in any civil action or administrative procedure. Page 84 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Note: GAO comments supplementing those in the report text appear at the end of this appendix. Page 85 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 86 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 87 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 88 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation See comment 1. See comment 2. Page 89 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 90 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation See comment 3. See comment 4. See comment 5. Page 91 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation See comment 6. See comment 7. Page 92 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation See comment 8. See comment 3. Page 93 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 94 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation See comment 9. Page 95 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation See comment 10. See comment 11. Page 96 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 97 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation See comment 12. Page 98 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 99 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 100 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation Page 101 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation The following are GAO’s comments on Chairman Brown and Commissioner Moore’s letter dated July 24, 1997. 1. As we discuss in the report, CPSC relies on a number of different sources GAO Comments for injury incident data. (Descriptive information on all CPSC’s data sources is provided in app. V.) However, only one source—the NEISS system—is capable of providing statistically reliable, representative information, and the NEISS system covers only those injuries treated in hospital emergency rooms. Although CPSC obtains information from other sources, these data are anecdotal and thus their usefulness is very limited in estimating injury prevalence. As a result, we have recommended that CPSC consult with experts both inside and outside the agency to prioritize its additional data needs and to explore the feasibility of options for obtaining these data. Changes over the past decade in the health care market, including the growth of ambulatory care, changes in reimbursement procedures, and improved health services research have the potential to make such additional data collection more feasible and less costly than it was a decade ago when some of these assessments were last made. 2. Figure 3 correctly states that CPSC does not obtain systematic surveillance data on injuries treated outside the emergency room. In the text of the report, we state that the number of injuries treated in each setting is unknown. We added a similar statement to the figure to emphasize this point. 3. In several places in their comments, Chairman Brown and Commissioner Moore refer to CPSC’s investigations of selected incidents to obtain more information. We discussed these investigations in our report. We believe that these investigations provide valuable information on causation, characteristics of accident victims, and hazard patterns, and we agree that some information gleaned from investigations is not obtainable from surveillance data. However, according to CPSC staff, this information is available to the agency only after a project is well under way, not at the initial stage of project selection. As a result, CPSC has little information on these important factors to assist in project selection, and evaluation of these criteria at the project selection stage is thus unavoidably speculative. 4. Descriptive information on this source of information was provided on in appendix V. Page 102 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation 5. For reasons we discussed in the report, relying on aggregate data (data not specific to a particular consumer product) to address the limitations of CPSC’s surveillance data is problematic. In addition, in the briefing packages we reviewed, the estimates of injury incidents usually referred only to NEISS cases and were not extended with modeling techniques; because these techniques were used so infrequently for this purpose, we did not assess their application in this report. 6. In referring to the agency’s monitoring of other aggregated sources of injury data, Chairman Brown and Commissioner Moore state that “Tracking of this type is sufficient to assure the adequacy of our data, as we use it.” We disagree. Examining trend information from other sources (without a rigorous application to CPSC’s own specific needs) is not sufficient to compensate for (or even measure the magnitude of) the limitations of CPSC’s injury surveillance data. 7. We believe this comment may reflect a misunderstanding of our point. We do not mean to suggest that CPSC incorrectly relies on NEISS to provide information on chronic illnesses. Rather, we are pointing out that CPSC has virtually no systematic data on chronic illnesses. As we stated in the report, on page 20, we agree with Chairman Brown and Commissioner Moore (and with the CPSC staff we interviewed) that such data are often difficult to obtain. However, chronic illness is listed as a criterion for CPSC project selection, and CPSC has little information to assist in applying this criterion. Accordingly, we are pleased with the statement by Brown and Moore that CPSC will consider additional surveillance methods to obtain more information on chronic illnesses. 8. We acknowledge in the report that CPSC uses other sources to supplement its death certificate data. However, our interviews with CPSC staff and our review of agency documents confirmed that death certificates are the most important source of death data for CPSC. In the briefing packages we reviewed, 80 percent of the calculations for numbers of deaths were based in whole or in large part on death certificate data, and death certificates were the sole source of death information in 36 percent of all CPSC briefing packages (a far greater percentage than for any other single source). Death data were usually reported in CPSC briefing packages with a lag of 2 or more years, especially when death certificates were the sole source of data. 9. We agree with Chairman Brown and Commissioner Moore that there is no single mold or perfect set of criteria for evaluating a cost-benefit Page 103 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation analysis, and we have revised the report to further emphasize this point. We believe that Chairman Brown and Commissioner Moore may have misunderstood the purpose of our evaluation questions. We did not “derive a particular methodology” for cost-benefit analysis, nor do we mean to suggest that CPSC should follow some “formula” for conducting analyses that does not leave room for competent professional judgment. However, to say that there is no perfect “formula” for cost-benefit analysis does not imply that all methodological choices are equally consistent with rigorous and comprehensive professional work. Although no litmus test exists for a “good” analysis, the professional literature offers some basic, minimum elements that are commonly used in evaluating cost-benefit analyses. These elements, which are based on the principles of transparency and completeness, are generally considered necessary, although not sufficient, for a good analysis. Although flexibility may be sometimes necessary in the assumptions or models underlying a cost-benefit analysis, the elements we used—including full disclosure of data limitations, sensitivity analysis, and incorporating all important costs and benefits—are appropriate to a wide range of situations. 10. Chairman Brown and Commissioner Moore are correct in stating that we evaluated 29 CPSC cost-benefit analyses that were completed between January 1, 1990, and September 30, 1996. We identified these 29 analyses as complete on the basis of CPSC’s statements—specifically, we considered a cost-benefit analysis to be complete only if an explicit comparison was made between aggregate costs and benefits. In addition, although the analyses may have been prepared at different stages of the project, we based our review on all available documentation on the project, and we reported results only for the evaluation questions that applied to the majority of cases and for which a clear determination could be made. We did not report results separately for the eight regulatory analyses that were required by law, because there were relatively few of these. However, we found no substantial differences between these 8 and the remaining 21 in terms of how they performed, compared with commonly used elements of evaluation of cost-benefit analysis. Therefore, we are confident that our results present an accurate assessment of CPSC’s cost-benefit analyses, and we recommend that the agency implement changes to ensure that its analyses are comprehensive and reported in sufficient detail. 11. We believe that, as now constructed, CPSC’s method for tracking projects operates at too high a level of generality and provides too little information to give a comprehensive, accurate picture of the agency’s activities either at a given point or over a longer period. CPSC staff told us, Page 104 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VI Comments From the Chairman and Commissioner Moore, Consumer Product Safety Commission, and GAO’S Evaluation and our review of agency documentation confirmed, that the Management Information System (MIS) usually tracks most agency activities only at a very general level. For example, CPSC’s 1996 year-end MIS report lists some specific projects such as “upholstered furniture” and “range fires,” but most projects are accounted for under either broad umbrella codes such as “sports and recreation” or “children’s projects,” or under activity codes such as “investigations,” “product safety assessment,” or “emerging problems.” In addition, CPSC staff told us that reliable inferences on resources spent cannot be drawn from MIS data because of limitations in the computer system and because no consistent rule exists about how staff time in different directorates is recorded to project codes. As a result, CPSC staff were unable to generate a comprehensive list of projects or to provide accurate information about resources allocated to those projects. We recommend an improved tracking system that would provide enough information to monitor the projects selected and resources spent for each specific consumer product hazard. We believe that as CPSC develops its planned accounting system, it should attempt to make it as compatible as practicable with the recommended tracking system. Nevertheless, we believe that whether or not it implements its planned accounting system, CPSC can and should improve its ability to track projects. 12. Our interviews with present and former commissioners revealed a pattern by which certain of CPSC’s regulatory criteria have historically been given greater emphasis in CPSC’s project selection process. Our objective was to describe the process as it was related to us; we have not taken a position on whether this process is appropriate. We have added a statement to our methodology section to emphasize this point. Page 105 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VII Comments From Commissioner Gall, Consumer Product Safety Commission, and GAO’S Evaluation Note: GAO comments supplementing those in the report text appear at the end of this appendix. See comment 1. Page 106 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VII Comments From Commissioner Gall, Consumer Product Safety Commission, and GAO’S Evaluation Page 107 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VII Comments From Commissioner Gall, Consumer Product Safety Commission, and GAO’S Evaluation See comment 2. See comment 3. Page 108 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VII Comments From Commissioner Gall, Consumer Product Safety Commission, and GAO’S Evaluation Page 109 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VII Comments From Commissioner Gall, Consumer Product Safety Commission, and GAO’S Evaluation The following are GAO’s comments on Commissioner Gall’s letter dated July 23, 1997. 1. Our interviews with present and former commissioners revealed a GAO Comments pattern by which certain of CPSC’s regulatory criteria have historically been given greater emphasis in CPSC’s project selection process. Our objective was to describe the process as it was related to us; we have not taken a position on whether this process is appropriate. We have added a statement to our methodology section to emphasize this point. 2. Commissioner Gall states that we did not analyze the relative importance of the deficiencies we found in CPSC’s data and methodology. However, as we stated in the report, available information does not permit us to determine the impact of better-quality data on the decisions CPSC made. The limitations we found in CPSC’s data have a variety of potentially conflicting impacts, precluding us from determining exactly how the results of the analysis might change if improved data were available. For example, because CPSC’s injury estimates are often confined to injuries treated in hospital emergency rooms, CPSC’s estimates will generally understate the actual number of injuries associated with a consumer product. However, CPSC’s systematic injury and death data can generally tell only whether a product was involved in an accident—not whether the product caused or contributed to the accident. As a result, this can make the risks assessed by CPSC appear larger than they might actually be. Similarly, we cannot determine how improved exposure data would change the relative importance of the risks assessed by CPSC. We agree with Commissioner Gall that not all projects will merit the same level of data or analysis. However, our review of CPSC raises questions about the agency’s ability to obtain and analyze data necessary to support rigorous analysis of important agency projects. 3. We agree with Commissioner Gall that resource considerations should enter into CPSC’s decisions to undertake new data collection. For this reason, we recommended an overall feasibility study for CPSC to prioritize among its data needs and investigate new options for obtaining additional information. Page 110 GAO/HEHS-97-147 Consumer Product Safety Commission Appendix VIII GAO Contacts and Staff Acknowledgments Lawrence J. Horinko, Assistant Director, (202) 512-7001 GAO Contacts Sarah L. Glavin, Project Manager, (202) 512-7180 In addition to those named above, the following individuals made Staff important contributions to this report: Sheila A. Nicholson, Analyst, Acknowledgments gathered and analyzed data on CPSC’s information release procedures; Nancy K. Kintner-Meyer, Senior Evaluator, compiled and analyzed information on CPSC projects; George Bogart, Senior Attorney, provided legal assistance; Harold Wallach, Senior Analyst, assisted in the analysis of CPSC’s data systems; and Charles Jeszeck served as Assistant Director for the project in its early stages. Page 111 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Abkowitz, M., and P. Der-Ming Cheng. “Developing a Risk/Cost Framework for Routing Truck Movements of Hazardous Materials.” Accident Analysis and Prevention, 20 (1988), pp. 39-51. Abraham, John. “Negotiation and Accommodation in Expert Medical Risk Assessment and Regulation: An Institutional Analysis of the Benoxaprofen Case.” Policy Sciences, 27 (1994), pp. 53-76. Adams, John. “The Emperor’s Old Clothes: The Curious Comeback of Cost-Benefit Analysis.” Environmental Values, 2 (1993), pp. 247-60. Adler, Robert S. “From ’Model Agency’ to Basket Case—Can the Consumer Product Safety Commission Be Redeemed?” Administrative Law Review, 42 (Winter 1989), pp. 61-129. Alberini, Anna. “Testing Willingness-to-Pay Models of Discrete Choice Contingent Valuation Survey Data.” Land Economics, 71 (1) (Feb. 1995), pp. 83-95. Allen, Julius. Cost Benefit Analysis of Federal Regulation: A Review and Analysis of Developments, 1978-1984. Report No. 84-74 E. Washington, D.C.: Congressional Research Service, May 15, 1984. Allison, Paul D. Event History Analysis: Regression for Longitudinal Event Data, Sage University Paper Series: Quantitative Applications in the Social Sciences, series no. 07-046. Newbury Park, Calif.: Sage Publications, 1984. American Academy of Pediatrics. “Hospital Discharge Data on Injury: The Need for E Codes.” Policy Statement in AAP News, Mar. 1992. Arrow, Kenneth J., and others. Benefit-Cost Analysis in Environmental, Health, and Safety Regulation: A Statement of Principles. American Enterprise Institute, The Annapolis Center, and Resources for the Future. Washington, D.C.: AEI Press, 1996. Asch, Peter. Consumer Safety Regulation: Putting a Price on Life and Limb. New York: Oxford University Press, 1988. Ault, Eric B. “CPSC’s Voluntary Standards: An Assessment and a Paradox.” The Frontier of Research in the Consumer Interest. Eds. E. Scott Maynes and ACCI Research Committee. Columbia, Mo.: American Council on Consumer Interests, 1986, pp. 77-81. Page 112 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Bailey, Martin J. Reducing Risks to Life: Measurement of the Benefits. Washington, D.C.: American Enterprise Institute, 1980. Baker, S.P., and others. The Injury Fact Book, 2nd ed. New York: Oxford University Press, 1992. Battiato, Salvatore Enrico. “Cost-Benefit Analysis and the Theory of Resource Allocation.” Alan Williams and Emilio Giardina. Efficiency in the Public Sector. Aldershot, Hants, England: Edward Elgar Publishing, 1993, pp. 26-42. Berlage, L., and R. Renard. “The Discount Rate in Cost-Benefit Analysis and the Choice of a Numeraire.” Oxford Economic Papers, 37 (1985), pp. 691-9. Berlau, John. “Play (Regulated) Ball “ Reason, Dec. 1996, pp. 71-3. Bhat, M., and S. Li. “Consumer-Product-Related Tooth Injuries Treated in Hospital Emergency Rooms: United States, 1979-1987.” Community Dentistry and Oral Epidemiology, 18 (3) (June 1990), pp. 133-8. Bird, Peter J.W.N. “A Note on Relative Price Changes in Cost-Benefit Analysis.” Public Finance, 35 (2) (1980), pp. 318-23. Blackorby, Charles, and David Donaldson. “A Review Article: The Case Against the Use of the Sum of Compensating Variations in Cost-Benefit Analysis.” Canadian Journal of Economics, 23 (3) (Aug. 1990), pp. 471-94. _____. “Welfare Ratios and Distributionally Sensitive Cost-Benefit Analysis.” Journal of Public Economics, 34 (1987), pp. 265-90. Boardman, Anthony, Aidan Vining, and W.G. Waters II. “Costs and Benefits Through Bureaucratic Lenses: Example of a Highway Project.” Journal of Policy Analysis and Management, 12 (3) (1993), pp. 532-55. Bordley, Robert F. “Making Social Trade-offs Among Lives, Disabilities, and Costs.” Journal of Risk and Uncertainty, 9 (1994), pp. 135-49. Boustead, Thomas. The Law and Economics of Administrative Law: A Statistical Analysis of the Consumer Product Safety Commission’s Petition Process. Ph.D. Dissertation, Fordham University, May 1995. Page 113 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Boyle, M.H., and others. “Economic Evaluation of Neonatal Intensive Care of Very Low Birthweight Infants.” New England Journal of Medicine, 308 (1983), pp. 1330-37. Bradfield, R.E. “Social Cost-Benefit Analysis in Project Evaluation: A Practical Framework for Decision Making.” Journal of Studies in Economics and Econometrics, 13 (2) (1989), pp. 25-35. Brent, Robert J. “The Cost-Benefit Analysis of Government Loans.” Public Finance Quarterly, 19 (1) (Jan. 1991), pp. 43-66. Breyer, Stephen. Breaking the Vicious Circle. Cambridge, Mass.: Harvard University Press, 1993. Burt, Catharine E. “Injury-Related Visits to Hospital Emergency Departments: United States, 1992.” Advance Data, No. 261 (Feb. 1, 1995). Buzby, Jean C., Richard C. Ready, and Jerry R. Skees. “Contingent Valuation in Food Policy Analysis: A Case Study of a Pesticide-Residue Risk Reduction.” Journal of Agricultural and Applied Economics, 27 (2) (Dec. 1995), pp. 613-25. Campen, James T. Benefit, Cost, and Beyond: The Political Economy of Benefit-Cost Analysis. Cambridge, Mass.: Ballinger Publishing Company, 1986. Cesario, Frank J. “Benefit-Cost Analysis Under Pricing Constraints.” Applied Economics, 13 (1981), pp. 215-24. Chan, Yuk-Shee, and Anthony M. Marino. “Regulation of Product Safety Characteristics Under Imperfect Observability.” Journal of Regulatory Economics, 6 (1994), pp. 177-95. Chirinko, Robert S., and Edward P. Harper, Jr. “Buckle Up or Slow Down? New Estimates of Offsetting Behavior and Their Implications for Automobile Safety Regulation.” Journal of Policy Analysis and Management, 12 (2) (1993), pp. 270-96. The Institute of Medicine and the National Research Council, Commission on Life Sciences, Committee on Trauma Research. Injury in America: A Continuing Public Health Problem. Washington, D.C.: National Academy Press, 1985. Page 114 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Cost-Benefit Analysis: Wonder Tool or Mirage? Report by the U.S. House of Representatives Subcommittee on Oversight and Investigation, Committee on Interstate and Foreign Commerce, Dec. 1980. Cowen, Tyler. “The Scope and Limits of Preference Sovereignty.” Economics and Philosophy, 9 (1993), pp. 253-69. Cox, Louis Anthony (Jr.). “Theory of Regulatory Benefits Assessment: Econometric and Expressed Preference Approaches.” Benefits Assessment: The State of the Art. Eds. Judith D. Bentkover, Vincent T. Covello, and Jeryl Mumpower. Dordrecht, Holland: D. Reidel Publishing Company, 1986, pp. 85-160. Crandall, Robert W. “The Use of Cost-Benefit Analysis in Product Safety Regulation.” The Frontier of Research in the Consumer Interest. Eds. E. Scott Maynes and ACCI Research Committee. Columbia, Mo.: American Council on Consumer Interests, 1986, pp. 61-75. Crews, Clyde Wayne. “Ten Thousand Commandments: Regulatory Trends 1981-92 and the Prospect for Reform.” Journal of Regulation and Social Cost, Vol. 2 (Mar. 1993), pp. 105-50. Cummings, Ronald G., Louis Anthony Cox, Jr., and A. Myrick Freeman III. “General Methods for Risk Assessment.” Benefits Assessment: The State of the Art. Eds. Judith D. Bentkover, Vincent T. Covello, and Jeryl Mumpower. Dordrecht, Holland: D. Reidel Publishing Company, 1986, pp. 161-92. Curtin, Leo, and Ronald Krystynak. “An Economic Framework for Assessing Foodborne Disease Control Strategies With an Application to Salmonella Control in Poultry.” The Economics of Food Safety. Ed. Julie A. Caswell. London: Elsevie, 1991, pp. 131-51. Dasguptha, A.K., and D.W. Pearce. Cost-Benefit Analysis: Theory and Practice. London: Macmillan, 1972. Davis, Miriam. Health Risk Assessment Research at the Department of Energy: Background Information. Washington, D.C.: U.S. Office of Technology Assessment, Sept. 15, 1992. Davis, Yvette, and others. “An Evaluation of the National Electronic Injury Surveillance System for Use in Monitoring Nonfatal Firearm Injuries and Page 115 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Obtaining National Estimates.” Journal of Safety Research, 27 (2) (Summer 1996), pp. 83-91. Dehar, Mary Anne, Sally Casswell, and Paul Duignan. “Formative and Process Evaluation of Health Promotion and Disease Prevention Programs.” Evaluation Review, 17 (2) (Apr. 1993), pp. 204-20. Drummond, Michael F. “Cost Benefit Analysis in Health and Health Care: Fine in Practice, but Does It Work in Theory?” Alan Williams and Emilio Giardina. Efficiency in the Public Sector. Aldershot, Hants, England: Edward Elgar Publishing, 1993, pp. 106-28. Drummond, Michael F., Greg L. Stoddart, and George W. Torrance. Methods for Economic Evaluation of Health Care Programmes. New York: Oxford University Press, 1996. Drummond, Michael F., and others. “Users’ Guide to the Medical Literature: How to Use an Article on Economic Analysis of Clinical Practice.” Journal of the American Medical Association, 277 (19) (May 21, 1997), pp. 1552-57. Evans, J.S., N.S. Hawkins, and J.D. Graham. “Uncertainty Analysis and the Value of Information: Monitoring for Radon in the Home.” Journal of Air Pollution Control Association, 38 (1988), pp. 1380-85. Evans, L., and M.C. Frick. “Helmet Effectiveness in Preventing Motorcycle Driver and Passenger Fatalities.” Accident Analysis and Prevention, 20 (1988), pp. 447-58. Fingar, Ann R., Richard S. Hopkins, and Marjorie Nelson. “Work-Related Injuries in Athens County, 1982 to 1986: A Comparison of Emergency Department and Workers’ Compensation Data.” Journal of Occupational Medicine, 34 (8) (Aug. 1992), pp. 779-87. Finkel, Adam M. Comparing Risks Thoughtfully. Washington, D.C.: U.S. Office of Technology Assessment, Sept. 1994. _____. “Who’s Exaggerating?” Discover, May 1996, pp. 45-54. Fischhoff, Baruch. Ranking Risks. Washington, D.C.: U.S. Office of Technology Assessment, Feb. 1994. Page 116 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography _____, and Louis Anthony Cox, Jr. “Conceptual Framework for Regulatory Benefits Assessment.” Benefits Assessment: The State of the Art. Eds. Judith D. Bentkover, Vincent T. Covello, and Jeryl Mumpower. Dordrecht, Holland: D. Reidel Publishing Company, 1986, pp. 51-84. Fise, Mary Ellen. “Consumer Product Safety Regulation,” Chapter 11 in Regulation and Consumer Protection. Dame Publications, Inc., 1995, pp. 277-94. _____. The CPSC: Guiding or Hiding From Product Safety? Washington, D.C.: Consumer Federation of America, May 1987. _____, and M. Kristen Rand. Pennies for Consumer Protection? A Report on the CPSC’s 1991 Budget. Washington, D.C.: Consumer Federation of America, May 1990. Fisher, Ann, Maria Pavlova, and Vincent Covello, eds. Evaluation and Effective Risk Communication Workshop Proceedings. Washington, D.C.: Interagency Task Force on Environmental Cancer and Heart and Lung Disease, Committee on Public Education and Communication, Jan. 1991. Fisher, Ann, Lauraine G. Chestnut, and Daniel M. Violette. “The Value of Reducing Risks of Death: A Note on New Evidence.” Journal of Policy Analysis and Management, 8 (1) (1989), pp. 88-100. Fuchs, Edward Paul, and James Anderson. “Institutionalizing Cost-Benefit Analysis in Regulatory Agencies.” Research in Public Policy Analysis and Management, 4 (1987), pp. 187-211. Fung, V.A., J.C. Barrett, and J. Huff. “Reviews and Commentaries: The Carcinogenesis Bioassay in Perspective: Application in Identifying Human Cancer Hazards.” Environmental Health Perspectives, 103 (1995), pp. 7-8. Gerner, Jennifer L. “Product Safety: A Review.” The Frontier of Research in the Consumer Interest, eds. E. Scott Maynes and ACCI Research Committee. Columbia, Mo.: American Council on Consumer Interests, 1986, pp. 37-59. Gillroy, John Martin, and Maurice Wade, eds. The Moral Dimensions of Public Policy Choice. Pittsburgh, Pa.: University of Pittsburgh Press, 1992. Page 117 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Gold, Marthe R., and others. Cost-Effectiveness in Health and Medicine. New York: Oxford University Press, 1996. Goldenhar, Linda, and Paul A. Schulte. “Intervention Research in Occupational Health and Safety.” Journal of Occupational Medicine, 36 (7), pp. 763-75. Goldstein, B. “Risk Management Will Not Be Improved by Mandating Numerical Uncertainty Analysis for Risk Assessment.” University of Cincinnati Law Review, Vol. 63, pp. 1599-1610. Goodstein, Eban. “Benefit-Cost Analysis at the EPA.” Journal of Socio-Economics, 24 (2) (1995), pp. 375-89. Graham, John D. “Making Sense of Risk: An Agenda for Congress.” Risks, Costs and Lives Saved: Getting Better Results from Regulation. Ed Robert W. Hahn. Washington, D.C.: AEI Press, 1996, pp. 183-207. _____. “Technology, Behavior, and Safety: An Empirical Study of Automobile Occupant-Protection Regulation.” Policy Sciences, 17 (1984), pp. 141-51. _____, and Younghee Lee. “Behavioral Response to Safety Regulation: The Case of Motorcycle Helmet-Wearing Legislation.” Policy Sciences, 19 (1986), pp. 253-73. Graves, Edmund J., and Brenda S. Gillum. “1994 Summary: National Hospital Discharge Survey.” Advance Data, No. 278 (Oct. 3, 1996). Gray, George M. The Challenge of Risk Characterization. Washington, D.C.: U.S. Office of Technology Assessment, Nov. 1992. Greenberg, David H. “Conceptual Issues in Cost/Benefit Analysis of Welfare-to-Work Programs.” Contemporary Policy Issues, 10 (Oct. 1992), pp. 51-64. Gregory, Robin, Thomas C. Brown, and Jack L. Knetsch. “Valuing Risks to the Environment.” The Annals of the American Academy of Political and Social Science. Eds. Howard Kunreuther and Paul Slovic. Vol. 545 (May 1996), pp. 54-63. Page 118 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Griffin, Ronald. “On the Meaning of Economic Efficiency in Policy Analysis.” Land Economics, 71 (1) (Feb. 1995), pp. 1-15. Guyer, Bernard, and Susan S. Gallagher. “An Approach to the Epidemiology of Childhood Injuries.” Pediatric Clinics of North America, 32 (1) (Feb. 1985), pp. 5-15. Haas, C.N., and others. “Risk Assessment of Viruses in Drinking Water.” Risk Analysis, Vol. 13, pp. 545-52. Hadden, Susan G. Read the Label: Reducing Risk by Providing Information. Boulder, Co.: Westview Press, 1986. Haddix, A.C., and others, eds. Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation. New York: Oxford University Press, 1996. Hahn, Robert W. “Improving Regulation: Steps Toward Reform.” Statement before the U.S. Senate Subcommittee on Financial Management and Accountability, Committee on Govermental Affairs, Sept. 25, 1996. _____. “A Preliminary Estimate of Some Indirect Costs of Environmental Regulation.” Unpublished paper, Feb. 1995. _____. “Regulatory Reform: A Legislative Agenda.” Statement before the U.S. Senate Committee on Governmental Affairs, Feb. 8, 1995. _____. “Regulatory Reform: What Do the Government’s Numbers Tell Us?” Risks, Costs and Lives Saved: Getting Better Results From Regulation. Ed. Robert W. Hahn. Washington, D.C.: AEI Press, 1996, pp. 208-54. _____. “Why We Need to Balance the Costs and Benefits of Regulation.” Statement before the U.S. House of Representatives, Subcommittee on Oversight and Investigations, Committee on Economic and Educational Opportunities, and the Subcommittee on Paperwork and Regulation, Committee on Small Business, Feb. 2, 1995. Hall, Margaret Jean, and Maria F. Owings. “Hospitalizations for Injury and Poisoning in the United States, 1991.” Advance Data, No. 252 (Oct. 7, 1994). Page 119 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Hammitt, J.K., and J.A.K. Cave. Research Planning for Food Safety: A Value of Information Approach. RAND R-3946-ASPE/NCTR. Santa Monica, Calif.: Rand, 1991. Hanke, Steve. “On the Feasibility of Benefit-Cost Analysis.” Public Policy, 29 (2) (Spring 1981), pp. 147-57. Haskins, Jack B. “Evaluative Research on the Effects of Mass Communication Safety Campaigns: A Methodological Critique.” Journal of Safety Research, 2 (2) (June 1970) pp. 86-8. Haupt, Barbara. “1982 Summary: National Hospital Discharge Survey.” Advance Data, No. 95 (Dec. 27, 1983). Havrilesky, Thomas. “The Persistent Misapplication of the Hedonic Damages Concept to Wrongful Death and Personal Injury Litigation.” Journal of Forensic Economics, 8 (1) (1995), pp. 49-54. Heimann, Christopher M., and others. “Project: The Impact of Cost-Benefit Analysis on Federal Administrative Law.” Administrative Law Review, 42 (Fall 1990), pp. 545-654. Henderson, J. Vernon. “Peak Shifting and Cost-Benefit Miscalculations.” Regional Science and Urban Economics, 22 (1992), pp. 103-121. Hensler, Deborah R., and others. Compensation for Accidential Injuries in the United States. Santa Monica, Calif.: RAND, The Institute for Civil Justice, 1991. Hildred, William, and Fred Beauvais. “An Instrumentalist Critique of ’Cost-Utility Analysis.’” Journal of Economic Issues, 29 (4) (Dec. 1995), pp. 1083-96. Hoehn, John P., and Alan Randall. “A Satisfactory Benefit Cost Indicator From Contingent Valuation.” Journal of Environmental Economics and Management, 14 (1987), pp. 226-47. _____. “Too Many Proposals Pass the Benefit-Cost Test.” American Economic Review, 79 (3) (June 1989), pp. 544-51. _____. “Too Many Proposals Pass the Benefit-Cost Test: Reply.” American Economic Review, 81 (5) (Dec. 1991), pp. 1450-52. Page 120 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Hoffer, George E., Stephen W. Pruitt, and Robert J. Reilly. “The Impact of Product Recalls on the Wealth of Sellers: A Reexamination.” Journal of Political Economy, 96 (3) (1988). Hopkins, Thomas D. “The Costs of Federal Regulation.” Journal of Regulation and Social Costs, Mar. 1992, pp. 5-31. Horowitz, John K., and Richard T. Carson. “Discounting Statistical Lives.” Journal of Risk and Uncertainty, 3 (1990), pp. 403-13. Hubin, Donald C. “The Moral Justification of Benefit-Cost Analysis.” Economics and Philosophy, 10 (1994), pp. 169-94. Jarrell, Gregg A., and Sam Peltzman. “The Impact of Product Recalls on the Wealth of Sellers.” George J. Stigler. Chicago Studies in Political Economy. Chicago, Ill.: Chicago Press, 1988, pp. 612-34. Johansson, Per-Olov. “Altruism in Cost-Benefit Analysis.” Environmental and Resource Economics, 2 (1992), pp. 605-13. Joksch, H. “Critique of Peltzman’s Study: The Effects of Automobile Safety Regulation.” Accident Analysis and Prevention, 8 (1976), pp. 213-14. Keeney, Ralph L. “The Role of Values in Risk Management.” The Annals of the American Academy of Political and Social Science. Eds. Howard Kunreuther and Paul Slovic. Vol. 545 (May 1996). Philadelphia: Sage Periodicals Press, pp. 126-34. Kemper, Peter, David A. Long, and Craig Thornton. “A Benefit-Cost Analysis of the Supported Work Experiment.” The National Supported Work Demonstration. Robinson G. Hollister, Peter Kemper, and Rebecca A. Maynard, eds. Madison, Wis.: University of Wisconsin Press, 1984, pp. 239-85. Kerton, Robert, and Richard Bodell. “Quality, Choice and the Economics of Concealment: The Marketing of Lemons.” Journal of Consumer Affairs, 29 (1) (Summer 1995), pp. 1-28. Kopp, Raymond J. “Why Existence Value SHOULD Be Used in Cost-Benefit Analysis.” Journal of Policy Analysis and Management, 11 (1) (1992), pp. 123-30. Page 121 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Langlois, Jean A., and others. Improving the E Coding of Hospitalizations for Injury: Do Hospital Records Contain Adequate Documentation?” American Journal of Public Health, 85 (9) (Sept. 1995), pp. 1261-65. Laughery, Kenneth R., David R. Lovvoll, and Meredith L. McQuilkin. “Allocation of Responsibility for Child Safety.” Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting. Santa Monica, Calif.: Human Factors and Ergonomics Society, 1996, pp. 810-13. Lave, Lester B. “Benefit Cost Analysis: Do the Benefits Exceed the Costs?” Risks, Costs and Lives Saved: Getting Better Results From Regulation. Ed. Robert W. Hahn. Washington, D.C.: AEI Press, 1996, pp. 104-34. Layard, Richard. Cost-Benefit Analysis. Middlesex, England: Penguin Books, 1972. Lescohier, Ilana, Susan S. Gallagher, and Bernard Guyer. “Not by Accident.” Issues in Science and Technology, Summer 1990, pp. 35-42. Lichtenberg, Erik. “Conservatism in Risk Assessment and Food Safety Policy.” The Economics of Food Safety. Ed. Julie A. Caswell. London: Elsevier, 1991, pp. 89-102. Linneman, Peter. “The Effects of Consumer Safety Standards: The 1973 Mattress Flammability Standard.” George J. Stigler. Chicago Studies in Political Economy. Chicago, Ill.: University of Chicago Press, 1988, pp. 441-60. Litan, Robert E., and William D. Nordhaus. Reforming Federal Regulation. New Haven: Yale University Press, 1983. Loomis, John B., and Pierre H. duVair. “Evaluating the Effect of Alternative Risk Communication Devices on Willingness to Pay: Results From a Dichotomous Choice Contingent Valuation Experiment.” Land Economics, 69 (3) (Aug. 1993), pp. 287-98. Lovvoll, David R., and others. “Responsibility for Product Safety in the Work Environment.” Proceedings of the Human Factors and Ergonomics Society 40th Annual Meeting. Santa Monica, Calif.: Human Factors and Ergonomics Society, 1996, pp. 814-17. MacLean, D. Values at Risk. Totowa, N.J.: Rowman and Allanheld, 1986. Page 122 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Magat, Wesley A., and Michael J. Moore. “Consumer Product Safety Regulation in the United States and the United Kingdom: the Case of Bicycles.” Rand Journal of Economics, 27 (1) (Spring 1996), pp. 148-64. Magat, Wesley A., and W. Kip Viscusi. Informational Approaches to Regulation. Cambridge, Mass.: MIT Press, 1992. Mayer, Michelle, and Felicia B. LeClere. “Injury Prevention Measures in Households with Children in the United States, 1990.” Advance Data, No. 250 (May 31, 1994). McCaig, Linda F. “National Hospital Ambulatory Medical Care Survey: 1992 Emergency Department Summary.” Advance Data, No. 245 (Mar. 2, 1994). McGarity, Thomas O. “Media Quality, Technology, and Cost Benefit Blancing Strategies for Health and Environmental Regulation.” Law and Contemporary Problems, 46 (3) (1983), pp. 159-233. _____, and Sidney A. Shapiro. Workers at Risk: The Failed Promise of the Occupational Safety and Health Administration. Westport, Conn.: Praeger Press, 1993. McGinnis, J.M., and W.H. Foege. “Actual Causes of Death in the United States.” Journal of the American Medical Association, Vol. 270, pp. 2207-12. McKean, Kevin. “They Fly in the Face of Danger.” Discover, Apr. 1986, pp. 48-58. Miller, James C. III, and Bruce Yandle, eds. Benefit-Cost Analyses of Social Regulation: Case Studies From the Council on Wage and Price Stability. Washington, D.C.: American Enterprise Institute, 1979. Miller, Ted R., and Diane C. Lestina. “Patterns in U.S. Medical Expenditures and Utilization for Injury, 1987.” American Journal of Public Health, 86 (1) (Jan. 1996), pp. 89-93. Mishan, E.J. Cost-Benefit Analysis. London: Unwin Hyman, 1988. Moore, John L. Cost Benefit Analysis: Issues in Its Use in Regulation. Report No. 95-760 ENR. Washington, D.C.: Congressional Research Service, June 28, 1995. Page 123 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Morgan, M. Granger, and Max Henrion. Uncertainty: A Guide to Dealing With Uncertainty in Quantitative Risk and Policy Analysis. Cambridge, Mass.: Cambridge University Press, 1990. Nash, Chris. “Cost-benefit Analysis of Transport Projects.” Alan Williams and Emilio Giardina. Efficiency in the Public Sector. Aldershot, Hants, England: Edward Elgar Publishing, 1993, pp. 83-105. National Committee for Injury Prevention and Control. Injury Prevention: Meeting the Challenge. New York: Oxford University Press, 1989. Needleman, Carolyn, and Martin L. Needleman. “Qualitative Methods for Intervention Research.” American Journal of Industrial Medicine, 29 (1996), pp. 329-37. Nelson, Cheryl R., and Barbara J. Stussman. “Alcohol- and Drug-Related Visits to Hospital Emergency Departments: 1992 National Hospital Ambulatory Medical Care Survey.” Advance Data, No. 251 (Aug. 10, 1994). Newbery, David. “The Isolation Paradox and the Discount Rate for Benefit-Cost Analysis: A Comment.” Quarterly Journal of Economics, Feb. 1990, pp. 235-8. Oi, Walter Y. “Safety at What Price?” American Economic Review, AEA Papers and Proceedings, 85 (2) (May 1995), pp. 68-71. Ozonoff, Victoria Vespe, Susan Tan-Torres, and Catherine W. Barber. “Assessment of E-Coding Practices and Costs in Massachusetts Hospitals.” Public Health Reports, 108 (5) (Sept. -Oct. 1993), pp. 633-6. Page, Talbot. “A Framework for Unreasonable Risk in the Toxic Substances Control Act (TSCA).” Management of Assessed Risk for Carcinogens. Ed. William J. Nicholson. Annals of the New York Academy of Sciences, New York: 1981. Page, Talbot, and Paolo F. Ricci. “A Cost-Benefit Perspective for Risk Assessment.” Principles of Health Risk Assessment. Ed. Paolo F. Ricci. Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1985, pp. 37-65. Pease, William S. The Impact of Scientific Research on Federal Agency Guidelines for Conducting Health Risk Assessments. Washington, D.C.: U.S. Office of Technology Assessment, Dec. 18, 1992. Page 124 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Peltzman, Sam. “The Effect of Automobile Safety Regulation.” George J. Stigler. Chicago Studies in Political Economy. Chicago, Ill. : University of Chicago Press, 1988, pp. 349-403 (originally published in Journal of Political Economy, 83, 1975, pp. 677-725). Petty, Ross D. “Bicycle Safety: A Case Study in Regulatory Review.” Regulation, 17 (2) (1994), pp. 22-24. _____. “Regulating Product Safety: The Informational Role of the U.S. Federal Trade Commission.” Journal of Consumer Policy, 18 (1995), pp. 387-415. Pollack, E.S., and D.G. Keimig, eds. Counting Injuries and Illnesses in the Workplace: Proposals for a Better System. New York: National Academy Press, 1987. Price, Colin. “Does Social Cost-Benefit Analysis Measure Overall Utility Change?” Economics Letters, 26 (1988), pp. 357-61. Proulx, Guylene, and Joelle Pineau. “The Impact of Age on Occupants’ Behavior During a Residential Fire.” Unpublished paper, 1996. Quiggin, John. “Existence Value and Cost-Benefit Analysis: A Third View.” Journal of Policy Analysis and Management, 12 (1) (1993), pp. 195-9. _____. “Too Many Proposals Pass the Benefit-Cost Test: Comment.” American Economic Review, 81 (5) (Dec. 1991), pp. 1446-9. Ramirez, Jorge, and others. “Ex-Post Analysis of Flood Control: Benefit-Cost Analysis and the Value of Information.” Water Resources Research, 24 (8) (Aug. 1988), pp. 1397-1405. Rauchschwalbe, Renae, and N. Clay Mann. “Pediatric Window-Cord Strangulations in the United States, 1981-1995.” Journal of the American Medical Association, 277 (21) (June 4, 1997), pp. 1696-98. Rice, Dorothy P., and Wendy Max. “Annotation: The High Cost of Injuries in the United States.” American Journal of Public Health, 86 (1) (Jan. 1996), pp. 14-15. Page 125 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Risa, Alf Erling. “Public Regulation of Private Accident Risk: The Moral Hazard of Technological Improvements.” Journal of Regulatory Economics, 4 (1992), pp. 335-46. Rivara, Frederick P., and others. “Cost Estimates for Statewide Reporting of Injuries by E Coding Hospital Discharge Abstract Data Base Systems.” Public Health Reports, 105 (6) (Nov.-Dec. 1990), pp. 635-38. Robins, Marcia P. “An Examination of the Effect of CPSC Prescription Drug Child-Resistant Packaging Requirements on Child Fatalities.” Unpublished paper, Nov. 1989. Rodgers, Gregory B. “The Effectiveness of Helmets in Reducing All-Terrain Vehicle Injuries and Deaths.” Accident Analysis and Prevention, 22 (1) (1990), pp. 47-58. _____. “Evaluating Product-Related Hazards at the Consumer Product Safety Commission: The Case of All-Terrain Vehicles.” Evaluation Review, Feb. 1990, pp. 3-21. _____, with P. Rubin. “Cost-Benefit Analysis of All-Terrain Vehicles at the CPSC.” Risk Analysis, 9 (1989), pp. 63-9. Rosenthal, Donald H., and Robert H. Nelson. “Why Existence Value Should NOT Be Used in Cost-Benefit Analysis.” Journal of Policy Analysis and Management, 11 (1) (1992), pp. 116-22. Rubin, Paul H., R. Dennis Murphy, and Gregg Jarrell. “Risky Products, Risky Stocks.” Regulation, No. 1 (1988), pp. 35-9. Rubinfeld, Daniel L., and Gregory B. Rodgers. “Evaluating the Injury Risk Associated With All-Terrain Vehicles: An Application of Bayes’ Rule.” Journal of Risk and Uncertainty, 5 (1992), pp. 145-58. Schappert, Susan M. “National Ambulatory Medical Care Survey: 1992 Summary.” Advance Data, No. 253 (Aug. 18, 1994). Schierow, Linda-Jo. “Risk Analysis and Cost-Benefit Analysis of Environmental Regulations. 94-961 ENR. Washington, D.C. : U.S. Congressional Research Service, Dec. 2, 1994. Schroeder, Elinor P., and Sidney A. Shapiro. “Responses to Occupational Disease: The Role of Markets, Regulation, and Information.” Georgetown Law Journal, 72 (1983), pp. 1231-1306. Page 126 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Selbst, S., M.D. Baker, and M. Shames. “Bunk Bed Injuries.” American Journal of Diseases in Children, 144 (6) (June 1990), pp. 721-23. Sen, Amartya. “Approaches to the Choice of Discount Rates for Social Benefit-Cost Analysis.” Chapter 8 in Resources, Values, and Development. Cambridge, Mass.: Harvard University Press, 1984, pp. 172-203. Senturia, Yvonne D., and others. “Exposure Corrected Risk Estimates for Childhood Product Related Injuries.” American Journal of Public Health, 1993, pp. 473-77. Shew, Russel, and Rachel Dardis. “An Economic Analysis of Child Restraints.” Journal of Consumer Policy, 18 (1995), pp. 417-31. Smith, V. Kerry. “A Conceptual Overview of the Foundations of Benefit-Cost Analysis.” Benefits Assessment: The State of the Art. Eds. Judith D. Bentkover, Vincent T. Covello, and Jeryl Mumpower. Dordrecht, Holland: D. Reidel Publishing Company, 1986, pp. 13-34. _____. “Uncertainty, Benefit-Cost Analysis, and the Treatment of Option Value.” Journal of Environmental Economics and Management, 14 (1987), pp. 283-92. _____, and others. “Can Public Information Programs Affect Risk Perceptions?” Journal of Policy Analysis and Management, 9 (1) (1990), pp. 41-59. Spitzer, Hugh. Case Studies to Demonstrate the Impact of Research on Assessment of Carcinogenic Risk. Washington, D.C.: U.S. Office of Technology Assessment, Dec. 1, 1992. Stason, W.B., and M.C. Weinstein. “Allocation of Resources to Manage Hypertension.” New England Journal of Medicine, 296 (1977), pp. 732-9. Stayner, Leslie, and others. “Approaches for Assessing the Efficacy of Occupational Health and Safety Standards.” American Journal of Industrial Medicine, 29 (1996), pp. 353-7. _____. “National Hospital Ambulatory Medical Care Survey: 1994 Emergency Department Summary.” Advance Data, No. 275 (May 17, 1996). Page 127 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Stussman, Barbara J. “National Hospital Ambulatory Medical Care Survey: 1993 Emergency Department Summary.” Advance Data, No. 271 (Jan. 25, 1996). Tengs, Tammy O., and John D. Graham. “The Opportunity Costs of Haphazard Social Investments in Life Saving.” Risks, Costs and Lives Saved: Getting Better Results from Regulation. Ed. Robert W. Hahn. Washington, D.C.: AEI Press, 1996, pp. 167-82. Thacker, Stephen B., and others. “Assessing Prevention Effectiveness Using Data to Drive Program Decisions.” Public Health Reports, 109 (2) (Mar.-Apr. 1994), pp. 187-94. Thomas, L.G. “Revealed Bureaucratic Preference: Priorities of the Consumer Product Safety Commission.” Rand Journal of Economics, 19 (1) (Spring 1988), pp. 102-13. Thompson, M.S., J.L. Read, and M. Laing. “Feasibility of Willingness-to-Pay Measuresment in Chronic Arthritis.” Medical Decision Making, 4 (2) (1984), pp. 195-215. Tolchin, Susan J., and Martin Tolchin. “Particles of Truth: Cotton Dust and Cost-Benefit Analysis.” Dismantling America: The Rush to Deregulate, Boston, Mass.: Houghton Mifflin Company, 1983, pp. 111-41. Tolley, G., K. Kenkel, and R. Rabian, eds. Valuing Health for Policy: An Economic Approach. Chicago, Ill.: University of Chicago Press, 1994. Trinkoff, Alison, and Peggy L. Parks. “Prevention Strategies for Infant Walker-Related Injuries.” Public Health Reports, 108 (6) (Nov.-Dec. 1993), pp. 784-8. Trumbull, William. “Reply to Whittington and MacRae.” Journal of Policy Analysis and Management, 9 (4) (1990), pp. 548-50. _____. “Who Has Standing in Cost-Benefit Analysis?” Journal of Policy Analysis and Management, 9 (2) (1990), pp. 201-18. U.S. Commission on Risk Assessment and Risk Management. Risk Assessment and Risk Management in Regulatory Decision-Making. Washington, D.C.: 1997. Page 128 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography U.S. Department of Health and Human Services, Centers for Disease Control and Prevention. “A Framework for Assessing the Effectiveness of Disease and Injury Prevention.” Morbidity and Mortality Weekly Report, Recommendations and Reports, Vol. 41, No. RR-3 (Mar. 27, 1992), pp. 1-13. _____. “Guidelines for Death Scene Investigation of Sudden Unexplained Infant Deaths.” Morbidity and Mortality Weekly Report, Recommendations and Reports, Vol. 45, No. RR-10 (June 21, 1996), pp. 1-31. _____. “Injuries and Deaths Associated With the Use of Snowmobiles—Maine, 1991-1996.” Morbidity and Mortality Weekly Report, Vol. 46, No. 1 (Jan. 10, 1997), pp. 1-4. _____. Inventory of Federal Data Systems in the United States for Injury Surveillance, Research and Prevention Activities. Washington, D.C.: May 1996. U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics. “Advance Report of Final Mortality Statistics, 1994.” Monthly Vital Statistics Report, 45 (3) (Supplement, Sept. 30, 1996). _____. “Hospitalizations for Injury and Poisoning in the United States, 1991.” Advance Data, No. 252 (Oct. 7, 1994). U.S. Department of Health and Human Services, Public Health Service. Healthy People 2000: Midcourse Review and 1995 Revisions. Washington, D.C.: 1995. _____. Healthy People 2000: National Health Promotion and Disease Objectives. Washington, D.C.: 1991. U.S. General Accounting Office. Peer Review: EPA’s Implementation Remains Uneven. Sept. 24, 1996, GAO/RCED-96-236. U.S. Office of Management and Budget. Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs. Circular A-94. Washington, D.C.: Oct. 29, 1992. _____. Regulatory Program of the United States Government. Bulletin No. 91-04. Washington, D.C.: Nov. 26, 1990. Page 129 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography U.S. Office of Technology Assessment. Researching Health Risks. OTA-BBS-570. Washington, D.C.: Jan. 1993. _____. Risks to Students in School. OTA-ENV-633. Washington, D.C.: Sept. 1995. U.S. Physician Payment Review Commission, A Comparison of Alternative Approaches to Risk Measurement. Washington, D.C.: Dec. 1994. van Ravenswaay, Eileen O., and John P. Hoehn. “The Impact of Health Risk Information on Food Demand: A Case Study of Alar and Apples.” The Economics of Food Safety. Ed. Julie A. Caswell. London: Elsevier, 1991, pp. 155-174. Veasie, M.A., and others. “Epidemiologic Research on the Etiology of Injuries at Work.” Annual Review of Public Health, 15 (1994), pp. 203-21. Viscusi, W. Kip. “The Dangers of Unbounded Commitments to Regulate Risk.” Risks, Costs and Lives Saved: Getting Better Results From Regulation. Ed. Robert W. Hahn. Washington, D.C.: AEI Press, 1996, pp. 135-66. _____. Regulating Consumer Product Safety. Washington, D.C.: American Enterprise Institute, 1984. _____. Tax Policy and the Economy. Ed. James M. Porterba, Vol. 9, 1995, p. 180. _____. “The Valuation of Risks to Life and Health: Guidelines for Policy Analysis.” Benefits Assessment: The State of the Art. Eds. Judith D. Bentkover, Vincent T. Covello, and Jeryl Mumpower. Dordrecht, Holland: D. Reidel Publishing Company, 1986, pp. 193-210. _____. “The Value of Risks to Life and Health. Journal of Economic Literature, 31 (Dec. 1993), pp. 1912-46. _____, and Richard J. Zeckhauser. “Hazard Communication: Warnings and Risk.” The Annals of the American Academy of Political and Social Science. Eds. Howard Kunreuther and Paul Slovic. Vol 545 (May 1996). Philadelphia: Sage Periodicals Press, pp. 106-15. Page 130 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography _____, and Wesley A. Magat, eds. Learning About Risk: Consumer and Worker Responses to Hazard Information. Cambridge, Mass.: Harvard University Press, 1987. Waller, Julian A. “Reflections on a Half Century of Injury Control.” American Journal of Public Health, 84 (4) (Apr. 1994), pp. 664-70. _____, Joan M. Skelly, and John H. Davis. “Treated Injuries in Northern Vermont.” Accident Analysis and Prevention, 27 (6) (1995), pp. 819-28. Warr, Peter G., and Brian D. Wright. “The Isolation Paradox and the Discount Rate for Benefit-Cost Analysis.” Quarterly Journal of Economics, 96 (1) (Feb. 1981), pp. 129-45. Weinstein, M.C., and others. “Recommendations of the Panel on Cost-Effectiveness in Health and Medicine.” Journal of the American Medical Association, 276 (15) (1996), pp. 1253-8. Weisbrod, Burton A. “Benefit-Cost Analysis of a Controlled Experiment: Treating the Mentally Ill.” Journal of Human Resources, 26 (4) (1981), pp. 523-48. Weiss, K.B., P.J. Gergen, and T.A. Hodgson. “An Economic Evaluation of Asthma in the United States.” New England Journal of Medicine, 326, pp. 862-6. Wheelwright, Jeff. “The Air of Ostrava: Pollution and Risk Assessment in the Czech Republic.” Discover, 17 (5) (May 1996), pp. 56-64. Whittington, Dale, and Duncan MacRae, Jr. “Comment: Judgments About Who Has Standing in Cost-Benefit Analysis.” Journal of Policy Analysis and Management, 9 (4) (1990), pp. 536-47. Wildasin, David E. “Indirect Distributional Effects in Benefit-Cost Analysis of Small Projects.” The Economic Journal, 98 (Sept. 1988), pp. 801-7. Williams, Alan. “Cost-Benefit Analysis: Applied Welfare Economics or General Decision Aid?” Alan Williams and Emilio Giardina. Efficiency in the Public Sector. Aldershot, Hants, England: Edward Elgar Publishing, 1993, pp. 65-82. Page 131 GAO/HEHS-97-147 Consumer Product Safety Commission Bibliography Wood, William W. “Cost-Benefit Analysis of Small Business Assistance: Do Entrepreneurs Really Need ’Assisting’?” Journal of Private Enterprise, 10 (1) (Summer 1994), pp. 13-21. Zeckhauser, Richard J., and W. Kip Viscusi. “The Risk Management Dilemma.” The Annals of the American Academy of Political and Social Science. Eds. Howard Kunreuther and Paul Slovic. Vol. 545 (May 1996). Philadelphia: Sage Periodicals Press, pp. 144-55. Zerbe, Richard O. “Comment: Does Benefit Cost Analysis Stand Alone? Rights and Standing.” Journal of Policy Analysis and Management, 10 (1) (1991), pp. 96-105. (205333) Page 132 GAO/HEHS-97-147 Consumer Product Safety Commission Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Consumer Product Safety Commission: Better Data Needed to Help Identify and Analyze Potential Hazards
Published by the Government Accountability Office on 1997-09-29.
Below is a raw (and likely hideous) rendition of the original report. (PDF)