United States General Accounting Office GAO Report to Congressional Committees October 1997 MAMMOGRAPHY SERVICES Impact of Federal Legislation on Quality, Access, and Health Outcomes GAO/HEHS-98-11 United States GAO General Accounting Office Washington, D.C. 20548 Health, Education, and Human Services Division B-276103 October 21, 1997 The Honorable James M. Jeffords Chairman The Honorable Edward M. Kennedy Ranking Minority Member Committee on Labor and Human Resources United States Senate The Honorable Thomas J. Bliley, Jr. Chairman The Honorable John D. Dingell Ranking Minority Member Committee on Commerce House of Representatives Breast cancer is the most commonly diagnosed nonskin cancer and the second leading cause of cancer deaths among American women. Experts estimate that during the 1990s as many as 1.8 million women will be diagnosed with breast cancer, and 500,000 will die from it. The probability of survival increases significantly, however, when the disease is discovered in its early stages. Currently, the most effective technique for early detection of breast cancer is screening mammography, an X-ray procedure that can detect small tumors and breast abnormalities up to 2 years before they can be detected by touch. The use of mammography as a tool for detecting early cancer continues to increase. The proportion of women aged 50 and older who had received mammograms in the past year increased from 26 percent in 1987 to 57 percent in 1995, according to the Centers for Disease Control and Prevention (CDC).1 The effectiveness of mammography as a cancer detection technique is directly related to the quality of mammography procedures. To address concerns about variations in the quality of mammography services provided by about 10,000 facilities throughout the United States and its territories, the Congress passed the Mammography Quality Standards Act of 1992 (MQSA). This act established a number of requirements aimed at strengthening mammography quality, including requiring accreditation and annual inspection of mammography facilities. The act is currently under congressional consideration for reauthorization. 1 The proportion of women 40 to 49 who had received mammograms in the past 2 years also increased: from 59 percent in 1990 to 66 percent in 1995. Page 1 GAO/HEHS-98-11 Mammography Services B-276103 The act also mandated that we assess and report on the program established by the Food and Drug Administration (FDA) to implement these requirements. We issued two interim reports, the first focusing on the act’s initial impact on access to and quality of mammography services, and the second focusing on FDA’s annual inspection program.2 As required by the act, this final report focuses on assessing the act’s effect on (1) the quality of mammography services, (2) early detection of breast cancer to save lives, and (3) women’s access to mammography services. We also followed up on the status of our previous recommendations to FDA (see app. II). This report is based primarily on our analysis of data obtained from FDA’s certification and annual inspection programs, as well as on research we reviewed regarding mammography’s effectiveness in cancer detection. We also consulted with many mammography and cancer experts during the course of our review. Details of our scope and methodology are presented in appendix I. Our work was done between January and August 1997 in accordance with generally accepted government auditing standards. MQSA has increased mammography facilities’ adherence to accepted quality Results in Brief assurance standards, which has, in turn, had a positive effect on mammography services. MQSA established nationwide minimum standards and required facility accreditation, which resulted in thousands of facilities’ having to improve their quality assurance processes. FDA’s annual inspections of facilities, now in their third year, continue to show increasing compliance with these national quality standards. Further evidence of quality improvement can be seen in the quality of the X-ray images. Before the act took effect, 11 percent of facilities tested were unable to pass image quality tests; now, the nationwide figure is 2 percent. Experts agree that improving the quality of mammography images should lead to more accurate interpretation by physicians and, therefore, to improved early detection of breast cancer. However, neither data nor research methodologies are now in place to clearly establish these links. FDA has established federal qualification requirements for physicians who interpret mammograms but has not established criteria for measuring interpretation accuracy. Furthermore, comparable pre- and post-MQSA clinical data for measuring mammography performance and cancer outcome either do not exist or, for a number of reasons, are too limited to be useful. Some steps are being taken to address interpretation accuracy 2 Mammography Services: Initial Impact of New Federal Law Has Been Positive (GAO/HEHS-96-17, Oct. 27, 1995) and FDA’s Mammography Inspections: While Some Problems Need Attention, Facility Compliance Is Growing (GAO/HEHS-97-25, Jan. 27, 1997). Page 2 GAO/HEHS-98-11 Mammography Services B-276103 and outcome measurement issues. FDA’s proposed final regulations require each facility to use its own data to monitor physicians’ performance on interpretation. In addition, as provided by MQSA, the National Cancer Institute (NCI) has established a Breast Cancer Surveillance Consortium of nine research projects. These projects are making progress in both developing a methodology and collecting clinical data for assessing trends in mammography performance in detecting breast cancer and reducing mortality. When MQSA was enacted, concern was expressed that some women might have difficulty obtaining mammography services if facilities chose to close down rather than to upgrade their operations to meet the new quality standards. We found no indication that access problems had developed as a result of MQSA. Nationwide, the number of facilities that stopped offering mammograms was nearly offset by the number of new entrants into the field. Further, 92 percent of all facilities that closed were within 5 miles of a facility that remained open, and our discussions with officials in states with the highest closure rates did not reveal any evidence that access problems had occurred. FDA has informed us that it has acted to implement our earlier recommendations for strengthening the MQSA inspection program. For example, FDA has efforts under way to establish procedures, guidance, and training to ensure timely compliance with MQSA standards. Our previous recommendations and the status of FDA actions taken on them appear in appendix II. Research studies, including randomized clinical trials, indicate that Background widespread use of mammography could reduce breast cancer mortality by one-third, especially in older women. The value of mammography in reducing mortality is directly tied to its ability to detect cancer at its earliest stages. Mammography is capable of detecting tumors much smaller than those detected by other means (see fig. 1).3 3 Mammography, however, is not a perfect tool; even under ideal conditions, 10 to 20 percent of breast cancers cannot be detected by mammography. Page 3 GAO/HEHS-98-11 Mammography Services B-276103 Figure 1: Size of Breast Tumors Found by Mammography and Self-Examination Source: Altered graphic from Choices, Marion Morra and Eve Potts (New York City: Avon Books, 1994), p. 303. Although mammography can be very useful in finding early-stage cancer, it is one of the most technically challenging radiological procedures, and ensuring the quality of the image is difficult. If the image is poor, tumors and abnormalities may go undetected. To illustrate, two images of the same patient who had a cancerous tumor are presented in figure 2. The tumor is visible in the picture on the right, where the image is of higher quality, but it is blurred and indecipherable in the picture on the left. Page 4 GAO/HEHS-98-11 Mammography Services B-276103 Figure 2: Low- and High-Quality Mammography Images of the Same Patient Source: FDA. Accurate interpretation of mammograms is equally as important as image quality. According to radiological experts, mammograms are the most difficult radiographic images to read, and misreading mammograms can have considerable consequences. A mammogram that is incorrectly interpreted as showing an abnormality could cause a woman to go through Page 5 GAO/HEHS-98-11 Mammography Services B-276103 unnecessary and costly follow-up procedures, such as ultrasound or biopsies. A mammogram that is read as normal when an abnormality is actually present could result in missed diagnosis of early lesions and delayed treatment, which could cost a woman’s life. MQSA established minimum national quality standards for mammography facilities and contained a number of provisions designed to ensure the quality of the image and its interpretation. Among other things, MQSA required that • FDA establish quality standards for mammography equipment, personnel, and practices; • all mammography facilities be accredited by an FDA-approved accrediting body once every 3 years and obtain a certificate from FDA in order to legally provide mammography services after October 1, 1994; and • all mammography facilities have an annual evaluation by a qualified medical physicist and an annual inspection by FDA-approved inspectors that includes a test of image quality. FDA implemented MQSA under interim regulations issued in December 1993. The quality standards of FDA’s interim regulations were substantially based on standards and an accreditation program developed in 1987 by the American College of Radiology (ACR), a private, nonprofit professional association of radiologists. Since early 1994, FDA has been working with the National Mammography Quality Assurance Advisory Committee to develop final regulations.4 On April 3, 1996, FDA published the proposed final regulations for public comment but as of September 1997 had not issued the final regulations. To assess MQSA’s effect on the quality of mammography services, we MQSA Has Had a analyzed the results of implementing three important quality assurance Positive Effect on the requirements of the act: accreditation review, annual facility inspections, Quality of and testing the quality of X-ray images. Both accreditation and annual inspection processes required by MQSA show that facilities are doing a Mammography better job of complying with quality standards, and image tests show that Services the quality of images has improved. 4 The committee is responsible for advising FDA on the appropriateness of quality standards for mammography facilities and accrediting bodies and for studying (1) the effect of MQSA on access to services in rural and health-professional-shortage areas, (2) the costs and benefits of compliance with MQSA, and (3) the sufficient number of medical physicists after Oct. 1, 1999, to ensure compliance with MQSA. As of September 1997, the committee had not submitted final reports to the Congress. Page 6 GAO/HEHS-98-11 Mammography Services B-276103 Accreditation Process Has MQSA required that all of the nation’s approximately 10,000 mammography Brought Improvement in facilities, regardless of location or setting, pass an accreditation review to Compliance With Quality ensure that they meet quality assurance requirements for equipment performance, radiation safety, personnel qualifications, and clinical image Standards quality, among other things. Before MQSA, between 37 and 44 percent of mammography units in the country met ACR’s quality assurance requirements by participating in ACR’s voluntary accreditation program. Since MQSA’s implementation, FDA has approved ACR and the states of Arkansas, California, and Iowa as official accrediting bodies. Because of ACR’s pre-MQSA involvement in establishing the voluntary accreditation program, it serves as the major accreditation body, responsible for more than 95 percent of the workload. MQSA’s quality standards and the related accreditation process have had a substantial effect on improving quality assurance activities. In an earlier report,5 we noted that when MQSA initially took effect, many mammography units did not meet the standards. For example, between October 1, 1994, and August 1, 1995, about 2,600 (35 percent) of the mammography units that sought ACR accreditation initially failed to meet accreditation requirements. While almost all of these units were eventually granted full accreditation, they first had to demonstrate that they had improved their quality assurance activities. ACR data suggest that the accreditation process continues to result in more facilities’ meeting quality assurance standards. For example, the percentage of facilities that passed ACR’s accreditation on the first attempt increased from 66 percent in 1995 to 82 percent in 1997.6 On-Site Inspections Show MQSA’s inspection requirement gives FDA another means to ensure that Continued Improvement in facilities comply with standards on a day-to-day operating level. While Compliance accreditation is generally a mail-in-process that involves the submission and review of application materials, annual inspections are conducted on 5 GAO/HEHS-96-17, Oct. 27, 1995. 6 The accreditation process under ACR allows facilities to go through two reviews. Facilities that fail the first accreditation review can correct deficiencies and resubmit their application materials for a second review. If a facility fails the second review, accreditation is denied. The Arkansas and Iowa accreditation bodies allow facilities to have more than two reviews before denying accreditation. While the state of California has been approved as an accreditation body, it contracts with ACR for a portion of accreditation review. Page 7 GAO/HEHS-98-11 Mammography Services B-276103 site, which allows inspectors to verify information provided during the accreditation process.7 As FDA entered its third year of annual inspections, inspection results continued to show improvement in facility compliance with MQSA quality standards. Both level 1 and level 2 violations, which FDA considers to be significant and for which it requires the facility to submit written plans for corrective actions, have dropped.8 As shown in figure 3, from fiscal year 1995 to fiscal year 1997, the proportion of inspected facilities whose highest-level violation was level 1 dropped from 3 percent to 1 percent, and the proportion whose highest-level violation was level 2 dropped from 20 percent to 12 percent.9 Most facilities with violations had only level 3 violations (considered by FDA to be minor deviations); these numbers also dropped during the 3-year period. The portion of facilities with no violations at all nearly doubled, from about 30 percent in 1995 to about 55 percent in 1997. 7 Annual inspections are generally performed by FDA-approved state inspectors. FDA has contracted with virtually all states and territories to conduct inspections. 8 Level 1 violations are those that can have the most detrimental effect on the quality of mammography services—for example, a facility’s having personnel who do not meet FDA’s minimum qualification standards. For level 1 violations, FDA issues a warning letter; the facility must respond in writing to FDA within 15 working days of receiving the warning letter. Level 2 violations are considered less significant—for example, a facility’s not having an evaluation of equipment by a medical physicist during the past 14 months—and do not generate a warning letter. However, the facility must respond in writing to FDA within 30 working days of receiving its inspection report. Level 3 violations—for example, a facility’s not having the required documentation of a quality control test—do not require a written response. 9 Under their contracts with FDA, many states began inspecting facilities in January 1995, but some did not begin until later. In fiscal year 1997, FDA began to manage its inspection workload and compare its inspection results by fiscal years. Results for 1997 are for facilities inspected during the first 9 months of the fiscal year. We have no reason to assume that the percentages will change greatly for facilities that will be inspected during the remainder of the year. Page 8 GAO/HEHS-98-11 Mammography Services B-276103 Figure 3: Comparison of Mammography Facilities’ Inspection Results, Fiscal Years 1995-97 Another way to assess whether facilities are improving their compliance is to consider the extent of repeat violations—violations found in one inspection and recurring in the next inspection. As of June 30, 1997, 8,619 facilities had been inspected at least twice. Facilities had a better record for not repeating the more severe violations than they did for the minor deviations. Less than 1 percent of facilities that had level 1 violations in the first year repeated one or more of the same violations in the second year, while about 17 percent of those with level 3 violations in the first year repeated one or more of the same violations in the second year. FDA considers level 3 violations to be minor findings. However, to reduce the incidence of repeat violations, and in response to a recommendation in our second report, FDA has revised its policy guidance to include procedures for addressing repeat level 3 violations. Page 9 GAO/HEHS-98-11 Mammography Services B-276103 More Facilities Are Passing One of the most important aspects of FDA’s annual inspection process is Tests of Image Quality testing the performance of a facility’s mammography system through what is called a “phantom image test.” This test involves taking an X-ray image of a plastic block containing 16 test objects to determine how many of the test objects can be seen on the image generated by the facility’s mammography equipment.10 To pass the evaluation, the phantom image must show at least the minimum number of test objects required by FDA.11 Because FDA was already conducting some phantom image testing before MQSA, it is possible to make general comparisons about image quality before and after the law took effect. Before MQSA, FDA, in a cooperative agreement with the Conference of Radiation Control Program Directors (a professional association of state and local radiation control program officials), conducted several Nationwide Evaluation of X-ray Trends (NEXT) surveys to evaluate the technical performance of mammography facilities. The phantom image scores from these pre-MQSA NEXT surveys are considered the baseline for comparison with the phantom image scores from post-MQSA annual inspections, because the standards for evaluating phantom images are essentially identical. NEXT survey data before MQSA implementation showed that phantom image scores were improving under ACR’s voluntary accreditation program. Data from the post-MQSA inspections show continued improvement in image quality, as indicated by the acceptable rates of phantom image scores. For example, the proportion of facilities with acceptable phantom image scores has remained at 98 percent since 1995, as compared with 89 percent in 1992. 10 The plastic block, which represents an average-sized compressed breast, contains a wax insert that holds 16 test objects—6 fibrous structures (fibers), 5 embedded microcalcifications (speck groups), and 5 different-sized tumor-like masses that simulate growths that could be cancerous. 11 To pass the phantom image test, at least 4 fibers, 3 speck groups, and 3 masses need to be visible in the phantom image. Page 10 GAO/HEHS-98-11 Mammography Services B-276103 Because MQSA can be linked to improved quality of mammography images, Data and a logical case can be made that MQSA has also helped improve Methodology mammography interpretation, thereby improving early breast cancer Problems Prevent detection and, ultimately, saving lives. However, neither data nor research methodologies are now in place to clearly demonstrate these links. One Measuring MQSA’s problem is that criteria have not been established to assess how reliably Impact on Saving images are being interpreted. Another is the lack of comparable clinical data12 for measuring any outcome changes that have occurred as a result Lives of the act. In addition, researchers still disagree on how to compute mammography performance and cancer outcome measures. In response to MQSA requirements, NCI has funded research and established a consortium that is making progress in collecting clinical data and resolving disagreements over how to measure mammography performance and cancer outcomes. MQSA’s Impact on the We were unable to determine if MQSA has had an effect on improving the Quality of Image accuracy and reliability with which mammograms are interpreted, because Interpretation Is Unknown criteria are not available to determine an appropriate measure. How to develop a quality assurance system for monitoring the accuracy with which mammograms are interpreted is a controversial issue in the medical community. Several academic studies have shown wide variation in the interpretation of the same mammogram by different radiologists. On one hand, some experts suggest that some type of peer review or proficiency test is needed to improve accuracy. On the other hand, FDA and others take the position that such measures are too difficult and costly to implement and that a self-monitoring, facility-based system of physician performance assessment is a better approach to achieve the goal of quality interpretation. FDA’s current approach reflects its preference for a facility-based monitoring system that tracks and reviews mammography and pathology results. Current MQSA regulations include a general requirement that each facility have a medical audit system in place to collect and review clinical data, which includes follow-up on positive mammograms (those identified as suspicious or highly suggestive of cancer) and their correlation to biopsy results. In proposed final regulations published in April 1996, FDA requires that at least one interpreting physician review this information annually and that data be analyzed both collectively for the facility and individually for each interpreting physician. The results are then to be 12 Clinical data include patient information such as demographics, risk factors, mammography results, and pathology results. Page 11 GAO/HEHS-98-11 Mammography Services B-276103 used to provide feedback to the interpreting physician as part of the facility’s quality assurance system. However, these regulations do not include standards for measuring physician performance. FDA officials stated that one of the main reasons for the general nature of FDA’s requirements is that there is no consensus in the medical and scientific community on the most desirable methodologies for arriving at such standards. Clinical Data Are Comparable clinical data for measuring outcome changes are not available Insufficient to Assess for a number of reasons. Some data are available from clinical trials, CDC’s MQSA’s Contribution to National Breast and Cervical Cancer Early Detection Program, a facility-based medical audit system, and cancer registries. However, none Mammography of these is an appropriate source to use in assessing MQSA’s effect on Performance and Cancer mammography performance and cancer outcomes. Outcomes Randomized clinical trials and other less comprehensive studies have demonstrated that regular screening mammography can significantly reduce breast cancer mortality. Such studies, while lending support to using mammography as a cancer detection technique, have not reported data that allow MQSA’s specific contribution to be assessed. For example, the only clinical trial in the United States—New York’s Health Insurance Plan Study—was conducted from 1963 to 1986 and was based on equipment, imaging, and interpreting techniques that are no longer applicable today.13 Also, although several community studies have shown the effectiveness of mammography screening in early detection, the results of each study were applicable to only one geographic area, and none of the studies was current enough for a pre- and post-MQSA comparison.14 CDC’sNational Breast and Cervical Cancer Early Detection Program, established in 1991 to provide screening services to medically underserved women, has collected a fairly large volume of mammography and cancer outcome data. However, the data focus on only a subset of the population 13 Four other major clinical trials have been undertaken in Sweden, two in the United Kingdom, and two in Canada. 14 For example, one of the largest community studies was conducted in Albuquerque, New Mexico. The study tracked mammography and pathology results of 87,433 patients from 1991 to 1993 to assess the performance of community mammography screening. The study showed that community mammography screening could detect breast cancer at early, treatable stages and that the distribution of cancer stages was similar to that seen in successful clinical trials. Page 12 GAO/HEHS-98-11 Mammography Services B-276103 and are too incomplete to yield a meaningful analysis of the effect of MQSA on cancer outcomes.15 The current MQSA requirement for a facility-based medical audit system also has numerous limitations that prevent its consideration in developing national measures of MQSA’s effectiveness. The fundamental limitation is that because the data collected in this process were not intended to serve as a mechanism for comparable reporting, little attention has been given to ensuring data and system consistency across facilities. Each facility decides how to collect and use the results as a quality control feedback tool, and FDA acknowledges that, as expected, there is wide variability in the data facilities include in their medical audits and in how they track results. Problems also exist with trying to use various regional, state, and local cancer registries to evaluate MQSA’s effects. These registries collect clinical data on breast cancer cases, including stage and tumor size of cancer detected. For example, the National Cancer Institute’s Surveillance, Epidemiology, and End Results Program (SEER) has nine cancer registries that have been in existence since 1973.16 Since 1992, CDC has also funded a national program that establishes or improves cancer registries in states that do not have registries such as those of the SEER program. While these registries contain clinical data that track the status of cancer cases, they do not collect data showing whether the cancer was detected by mammography. Thus, they cannot be used to measure mammography performance over time. Furthermore, while SEER cancer registry data (which contain substantial historical data) can be used to evaluate the trends of cancer incidence, cancer detection, and mortality, they are not current enough to enable a pre- and post-MQSA implementation comparison. For example, MQSA inspection of facilities began in 1995 and, at the time of our review, the SEER registry data were only current through 1994. 15 CDC’s program works with state public health agencies to provide screening, referral, and follow-up services for underserved women. The program began with 6 states in 1991 and is currently operating in 35 states. We attempted to use CDC data to compare the cancer detection rates of the program before and after MQSA, but several key data elements, such as history of prior mammograms and stage of cancers, were too incomplete to be useful for our analysis. 16 The program consists of nine regional population-based cancer registries covering about 14 percent of the U.S. population. Page 13 GAO/HEHS-98-11 Mammography Services B-276103 No Clear Consensus Exists In addition to the problems involved with the lack of data, controversy on Appropriate Measures exists over how to measure mammography performance and related for Mammography cancer outcomes. Performance and Cancer Outcomes Mammography Performance Several performance measures are based on assessments of the extent to Measures which mammography results are accurate. The two most common measures involve sensitivity and specificity.17 Sensitivity is defined as the probability of detecting cancer when a cancer really exists, and specificity is defined as the probability of obtaining a negative mammogram when no cancer exists. The computation of these measures is shown in the following formulas: Sensitivity = True Positive/(True Positive + False Negative) Specificity = True Negative/(True Negative + False Positive) Although the definitions of sensitivity and specificity are widely accepted, calculating sensitivity and specificity is complicated by the need to know the number of true and false positives, as well as the number of true and false negatives (see table 1 for definitions). Table 1: Definitions of True and False Positive and Negative Classification of mammogram result Definition True positive Cancer diagnoseda during the specified follow-up period after a mammogram was interpreted as positive False positive No cancer diagnosed during the specified follow-up period after a mammogram was interpreted as positive True negative No cancer diagnosed during the specified follow-up period after a mammogram was interpreted as negative False negative Cancer diagnosed during the specified follow-up period after a mammogram was interpreted as negative a Cancer is generally considered present only after being confirmed by a pathological diagnosis. 17 Another type of performance measure uses the positive predictive values of mammograms (the proportion of women with a positive mammogram who are found to actually have breast cancer). This measure can be used with three separate definitions: it can be based on the number of cancers detected from positive mammograms, the number of biopsies recommended, and the number of biopsies performed. Using the second and third definitions would require information to be readily available about the number of positive mammograms that resulted in biopsy recommendations and the number of recommendations that actually resulted in biopsies’ being performed. Page 14 GAO/HEHS-98-11 Mammography Services B-276103 Moreover, while researchers generally accept the above definitions, they have not agreed on the proper length of time for the follow-up period. For example, researchers have used various follow-up time periods, ranging from 7 months to over a year. Using different time periods can result in different determinations of true or false positives and true or false negatives. For example, if a mammogram is interpreted as negative and the follow-up period is 12 months, then the negative mammogram is considered a false negative if a cancer is detected during the next 12 months. However, if the specified follow-up period is 7 months, then the negative mammogram is considered a false negative only if a cancer is detected within 7 months. In addition, researchers disagree on how certain mammography results should be classified. In 1992, the ACR developed the Breast Imaging Reporting and Data System, which established six discrete categories for interpreting physicians to use in recording the results of mammograms: “needs additional evaluation,” “negative,” “benign finding,” “probably benign finding,” “suspicious finding,” and “highly suggestive of malignancy.”18 Although researchers generally agree that two categories (“negative” and “benign finding”) should be considered to be negative mammograms and that two other categories (“suspicious finding” and “highly suggestive of malignancy”) should be considered positive mammograms, they disagree on how the remaining categories (“needs additional evaluation” and “probably benign finding”) should be classified.19 Standards for the follow-up time period and mammography result categories are essential for consistent application of such performance measures. Cancer Outcome Measures Because the ultimate goal of mammography is to reduce breast cancer mortality, some studies have used mortality reduction as a measure of mammography effectiveness in cancer outcomes. However, because women with breast cancer generally survive longer than 5 years, measuring the change in mortality reduction requires a long follow-up time. This need for a long follow-up period makes it difficult to use 18 Although many interpreting physicians are using these categories, they are not required to do so under MQSA. However, in FDA’s proposed final regulations, recording mammography results using these categories will be required for standardization purposes. 19 The “probably benign” category is often associated with a recommendation for another mammogram within 6 months. Page 15 GAO/HEHS-98-11 Mammography Services B-276103 mortality reduction as a measure for assessing MQSA’s effect on cancer outcomes, since not enough time has elapsed since MQSA’s implementation. As an alternative to mortality reduction rates, some researchers have used the early-stage disease rate as an intermediate measure of the effectiveness of mammography.20 This intermediate measure is thought to be appropriate because early detection of breast cancer has been shown to improve survival rates. However, considerable disagreement exists about how to define early-stage cancer for measurement purposes. Perhaps the most controversial problem relates to whether or not cases of carcinoma in situ (CIS) should be included when measuring mammography effectiveness. CIS is considered the earliest stage of breast cancer and is confined to the place or site where it started. In recent years, as the number of CIS cases detected by mammography has continued to increase, researchers have questioned how to classify them. Some experts view CIS as a very slow-growing, noninvasive tumor and contend that including CIS cases as early stages of cancer will overstate the benefit of mammography in reducing mortality, since a large proportion of CIS tumors never spread. Others argue that some CIS tumors can grow quickly and develop into serious, even fatal, disease and, therefore, CIS cases should be considered early-stage cancer. Because of the above controversies, some researchers now believe that the late-stage disease rate is a more appropriate intermediate measure than the early-stage disease rate. However, there is no clear consensus on the use of this measure either. Consortium Is Making To help evaluate mammography’s overall impact, MQSA called for a system Progress in Resolving Data that would collect a large volume of data on mammography results and and Measurement cancer detection from select geographical areas. In response, NCI established the Breast Cancer Surveillance Consortium in 1994, which Problems consists of nine federally funded research projects at sites in most of the major geographical regions of the United States.21 These projects collect data from affiliated area mammography practices and link mammography data to cancer registry data. The Consortium’s goal is to ensure that the data collected can be analyzed to address issues relating to mammography performance and cancer outcomes. By the year 2000, the Consortium 20 The most widely used approach to categorizing cancers involves the use of staging classifications, with the size of the tumor being one of the most important factors in determining the stage of the cancer. 21 These projects are being administered by the University of California at San Francisco, the Colorado Department of Public Health and the Environment, the Norris Cancer Center, the Fred Hutchinson Cancer Research Center, the Group Health Cooperative of Puget Sound, the University of New Mexico, the University of North Carolina, the University of Vermont, and the University of Iowa. Page 16 GAO/HEHS-98-11 Mammography Services B-276103 expects to have data on more than 3.2 million mammography examinations and more than 24,000 breast cancer cases.22 According to Consortium members, the large sample is necessary to examine mammography performance and cancer outcomes in different health care delivery systems and regions of the country and in populations of diverse race and socioeconomic status. To create an effective database, the Consortium is taking the following steps to overcome the kinds of data and measurement problems discussed above. • With regard to collecting clinical data, the Consortium has spent considerable time developing a consensus on which data elements to collect in order to obtain consistent and reliable data. The Consortium has been using standardized data collection procedures and linkage mechanisms to pool data from individual projects, but 1996 was the first year common data elements were collected at all projects. The Consortium has just begun some descriptive analysis using pooled data from all projects; however, analysis of cancer outcomes will not begin until the winter of 1998, because valid data on cases of breast cancer will not be available until then. • The Consortium is also working to obtain a consensus on performance measures such as sensitivity and specificity. Officials of the Consortium informed us that they had just begun to test aspects of these performance measures and could not tell us when they will be able to reach a consensus. These efforts can help strengthen the quality of information about the general efficacy of mammography in improving cancer outcomes, but they are unlikely to provide substantial information about MQSA’s specific effect because data are not available for a pre- and post-MQSA comparison. However, the Consortium efforts may have other positive MQSA-related benefits in that researchers involved with several of the nine research projects are currently helping mammography facilities collect data for cancer outcomes audits to meet MQSA requirements. When MQSA was passed, the Congress was concerned that access to MQSA Did Not Limit mammography services might be limited because many providers would Access choose to drop mammography services rather than upgrade operations to 22 Studies show that the incidence of breast cancer ranges from 2 to 10 cancers per 1,000 mammography examinations. Page 17 GAO/HEHS-98-11 Mammography Services B-276103 comply with the standards. This has not occurred. Facility closures, both in anticipation of MQSA and since MQSA took effect, appear to have had no adverse effect on access to mammography services. We addressed the question of the initial closures in our 1995 report. In all, 404 facilities, or about 4 percent of the approximately 10,000 facilities that were providing mammography services before MQSA was implemented, had ceased providing mammography services during the 12 months before MQSA became effective. On the basis of (1) a study conducted by a private research firm under contract with FDA and (2) our interviews with state officials, we concluded that these initial closures had no negative impact on access to services. To assess the impact of facility closures since MQSA implementation, FDA asked the same contractor to do an updated study. The study, completed in May 1997, found that 1,085 facilities had stopped offering mammography services either temporarily or permanently between December 15, 1994, and March 19, 1997. During the same period, 922 facilities had opened or had resumed providing services. This net loss of 163 facilities since MQSA was implemented is a relatively small number when compared with the approximately 10,000 facilities operating before MQSA. Further, 99 percent of the closed facilities were located within 25 miles of another certified mammography facility, and 92 percent were within 5 miles. To determine if the states most affected by these closures had experienced problems with access, we conducted additional follow-up work on the contractor’s study. For the District of Columbia and the seven states identified in the contractor’s study as having closure rates of at least 4 percent, we asked state health officials to determine if the closure of any facility on the list had caused access problems. No problems were reported to us. Officials said many of the closed facilities were either low-volume providers that did not generate enough revenue to cover the costs of meeting MQSA requirements or poor-quality providers that could not pass accreditation. Furthermore, in five of the seven states, more than 40 percent of the facilities identified as closed have actually continued to provide services in some form. For example, some facilities continued to provide services by merging with another facility or consolidating their equipment from satellite clinics. Some other facilities contracted with mobile service providers to continue serving their patients. Page 18 GAO/HEHS-98-11 Mammography Services B-276103 As the Congress considers reauthorization of MQSA, two points are clear Conclusions with regard to assessing what the act has accomplished. First, overall, MQSA has had a positive impact on the quality of mammography services and no effect on access to them. In looking at the currently measurable areas, such as the accreditation and inspection results, the quality of X-ray images, and the extent of facility closures, the evidence is strong that the quality of services has improved and that access has not been adversely affected. We believe it is reasonable to attribute a large part of the quality improvement to (1) MQSA processes that enforced accreditation standards that were not previously followed by many facilities and (2) FDA’s annual inspection process, which provides a valuable, systematic means of helping ensure that these higher standards are maintained. Second, quantifying MQSA’s effect on the accuracy of mammogram interpretation or on the improvement in cancer outcomes is much more problematic. Although data collection efforts now under way will probably make it easier to monitor the quality of image interpretation and the effects of mammography on cancer outcomes in the future, the absence of pre-MQSA data means that analysts may not be able to fully measure how the act itself has affected image interpretation and cancer outcomes. In commenting on a draft of this report, Department of Health and Human Agency Comments Services (HHS) officials agreed with our presentation of the issues. In addition, ACR officials provided some technical comments, which we incorporated as appropriate. Appendix III contains the full text of HHS’ comments. Page 19 GAO/HEHS-98-11 Mammography Services B-276103 We are sending copies of this report to the Secretary of Health and Human Services, the Commissioner of the Food and Drug Administration, the Director of the National Cancer Institute, the Director of the Centers for Disease Control and Prevention, the Director of the Office of Management and Budget, and other interested parties. We will also make copies available to others on request. Please contact me at (202) 512-7719 if you or your staff have any questions. Other GAO contacts and staff acknowledgments are listed in appendix IV. Bernice Steinhardt Director, Health Services, Quality and Public Health Issues Page 20 GAO/HEHS-98-11 Mammography Services Page 21 GAO/HEHS-98-11 Mammography Services Contents Letter 1 Appendix I 24 Scope and Methodology Appendix II 26 Follow-Up on Our January 1997 Report Appendix III 28 Comments From the Department of Health and Human Services Appendix IV 29 GAO Contacts and Staff Acknowledgments Appendix V 30 Acknowledgments Tables Table 1: Definitions of True and False Positive and Negative 14 Table II.1: Summary of Recommendations and FDA Actions 27 Figures Figure 1: Size of Breast Tumors Found by Mammography and 4 Self-Examination Figure 2: Low- and High-Quality Mammography Images of the 5 Same Patient Figure 3: Comparison of Mammography Facilities’ Inspection 9 Results, Fiscal Years 1995-97 Page 22 GAO/HEHS-98-11 Mammography Services Contents Abbreviations ACR American College of Radiology CDC Centers for Disease Control and Prevention CIS carcinoma in situ FDA Food and Drug Administration HHS Department of Health and Human Services MQSA Mammography Quality Standards Act of 1992 NCI National Cancer Institute NEXT Nationwide Evaluation of X-ray Trends SEER Surveillance, Epidemiology, and End Results Program Page 23 GAO/HEHS-98-11 Mammography Services Appendix I Scope and Methodology To assess whether the Food and Drug Administration’s (FDA) implementation of the Mammography Quality Standards Act of 1992 (MQSA) has resulted in improvement in the quality of mammography services, we relied on three analyses: the American College of Radiology’s (ACR) accreditation results, FDA’s annual inspection results, and FDA’s pre-MQSA surveys of mammography equipment performance. Specifically, we obtained and reviewed data on ACR’s accreditation failure rates and reasons for failures for each fiscal year from 1995 to 1997. In addition, we analyzed the results of FDA’s annual inspections that took place between January 1995 and June 1997 at about 10,000 mammography facilities nationwide. To assess improvement in the quality of X-ray images, we compared pre- and post-MQSA phantom image scores using data from the Nationwide Evaluation of X-ray Trends (NEXT) surveys and FDA’s annual inspections. The NEXT surveys collected data on the technical performance of mammography, including such things as phantom image scores, radiation dose, and existence of darkroom fog (excessive light that fogs the image). We considered the phantom image scores from pre-MQSA NEXT surveys to be the baseline for comparison with the phantom image scores from post-MQSA annual inspections. Although we did not perform a reliability assessment of FDA’s data systems, we conducted several data tests and found FDA’s data to be adequate to meet our objectives. To assess whether MQSA has improved the early detection of breast cancer, we reviewed numerous studies and analyses on the accuracy of physician interpretation, breast cancer incidence and mortality, and mammography’s effectiveness in cancer detection. We also interviewed many mammography and cancer experts; researchers; and officials from FDA, the National Cancer Institute (NCI), and the Centers for Disease Control and Prevention (CDC) to discuss mammography performance and cancer outcome measures, as well as the availability of current and future data to assess performance and outcome improvement. In addition, we consulted with members of NCI’s Breast Cancer Surveillance Consortium who provided a written response to our questions on the methodological and data problems in measuring mammography performance and assessing MQSA’s impact on cancer outcomes. To assess MQSA’s effect on accessibility, we examined an FDA contractor’s study on newly opened and newly closed mammography facilities, by state, between December 1994 and March 1997. We supplemented this work by interviewing state radiation control program officials in seven states—California, Colorado, Florida, Kansas, New Hampshire, North Dakota, and Virginia—and the District of Columbia to discuss why Page 24 GAO/HEHS-98-11 Mammography Services Appendix I Scope and Methodology facilities in their localities closed and the impact that these closures had on access to mammography services. Page 25 GAO/HEHS-98-11 Mammography Services Appendix II Follow-Up on Our January 1997 Report In our January 1997 report reviewing FDA’s mammography inspection program,23 we identified several issues that needed management attention with regard to conducting inspections and following up on deficiencies. FDA generally agreed with our recommendations and has taken action on all of them. The following table summarizes the problems, our recommendations, and the actions that FDA has taken in response. 23 GAO/HEHS-97-25, Jan. 27, 1997. Page 26 GAO/HEHS-98-11 Mammography Services Appendix II Follow-Up on Our January 1997 Report Table II.1: Summary of Recommendations and FDA Actions Problems we found Our recommendations FDA’s actions Inspection results varied Monitor the inspection FDA distributed new considerably from state to results more closely to inspection software to state. It was not clear ensure consistent reporting monitor inspection results whether the differences were of violations and corrections. more closely. FDA also due to variations in the conducted additional quality of mammography at training of staff who audit different facilities or the the performance of MQSA inspectors’ approaches in inspectors to ensure conducting inspections and consistent application of reporting results. inspection procedures. We found variability in how Strengthen procedures for FDA implemented an inspectors scored phantom assessing image quality by annual program to test images. providing additional MQSA inspectors on inspector training and phantom image scoring. guidance. FDA had no procedure in Protect patients from risk of FDA established a policy place for clinical image poor mammograms by that allows it to initiate a review or patient notification requiring follow-up clinical clinical image review and when evidence suggested image reviews and patient patient notification when problems with the quality of a notification when serious violations are found. facility’s mammograms. inspections detect serious violations. When violations posed a Develop procedures for (1) FDA issued guidance to serious health risk, FDA did determining when the compliance officers to not have criteria to help health risk is serious establish the criteria and determine when to require enough to justify the procedures necessary to the immediate suspension of immediate suspension of a determine the degree of a facility’s certificate. facility’s certificate and (2) health risks and when to implementing the seek suspension of a suspension. facility’s certificate. FDA’s follow-up efforts did Reevaluate the FDA revised its policy not always ensure that classification and guidance to strengthen its corrective actions were taken enforcement policy for level procedures for addressing on less serious violations. 3 violations. less serious violations that recur from one inspection to another. FDA’s compliance system Make complete, up-to-date FDA distributed an was inadequate in three field information on violations automated compliance offices included in our review accessible to compliance information system that because staff did not have personnel. provides current information direct access to inspection on inspection violations. databases. Page 27 GAO/HEHS-98-11 Mammography Services Appendix III Comments From the Department of Health and Human Services Page 28 GAO/HEHS-98-11 Mammography Services Appendix IV GAO Contacts and Staff Acknowledgments Frank Pasquier, Assistant Director, (206) 287-4861 GAO Contacts Sophia Ku, Evaluator-in-Charge, (206) 287-4888 In addition to those named above, the following individuals made Staff important contributions to this report: Donna Bulvin, Senior Evaluator; Acknowledgments Susan Lawes, Senior Social Science Analyst; Stan Stenersen, Senior Evaluator; Evan Stoll, Computer Specialist; and Craig Winslow, Senior Attorney. Page 29 GAO/HEHS-98-11 Mammography Services Appendix V Acknowledgments We would like to acknowledge the members of the NCI’s Breast Cancer Surveillance Consortium and officials from CDC’s Division of Cancer Prevention and Control for their advice and technical assistance on the methodological and data problems of measuring mammography performance and cancer outcomes. (108310) Page 30 GAO/HEHS-98-11 Mammography Services Ordering Information The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent. Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists. For information on how to access GAO reports on the INTERNET, send an e-mail message with "info" in the body to: firstname.lastname@example.org or visit GAO’s World Wide Web Home Page at: http://www.gao.gov PRINTED ON RECYCLED PAPER United States Bulk Rate General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. G100 Official Business Penalty for Private Use $300 Address Correction Requested
Mammography Services: Impact of Federal Legislation on Quality, Access, and Health Outcomes
Published by the Government Accountability Office on 1997-10-21.
Below is a raw (and likely hideous) rendition of the original report. (PDF)