United States General Accounting Office GAO Report to Congressional Requesters January 2003 DECENNIAL CENSUS Methods for Collecting and Reporting Data on the Homeless and Others without Conventional Housing Need Refinement GAO-03-227 a January 2003 DECENNIAL CENSUS Methods for Collecting and Reporting Highlights of GAO-03-227, a report to Data on the Homeless and Others Congressional Requesters Without Conventional Housing Need Refinement The Bureau of the Census The Bureau's original plan for releasing Service-Based Enumeration data partnered with local governments, was outlined in an April 1999 internal memorandum that called for the advocacy groups, and other separate release of data on people counted at "emergency and transitional organizations to help it enumerate shelters." The Bureau planned to combine other components of Service- people without conventional Based Enumeration, including people counted at soup kitchens, regularly housing. Counting this population—which includes shelter scheduled mobile food vans, and certain outdoor locations, into a single residents and the homeless—has category. Driving the Bureau's decision was its experience during the 1990 been a longstanding challenge for Census when it released separate counts of people found at shelters, on the the Bureau. A number of street, and similar locations that proved to be incomplete. The Bureau also organizations put substantial tried to ensure that the Service-Based Enumeration figures could not be used resources into an operation the as a "homeless" count, because it was not designed to provide a specific Bureau called Service-Based count of the homeless. Instead, the operation was part of a larger effort to Enumeration. In return, some count people without conventional housing. expected the Bureau to provide data that would help them plan and The Homeless Are Hard to Enumerate deliver employment, health, and other services. However, the Bureau did not release the data as planned, which raised questions about the Bureau’s decision- making on data quality issues. In response to a congressional request, GAO examined the Bureau’s decision-making process behind its change in plans. Source: U.S. Bureau of the Census. The Secretary of Commerce should In January 2001, the Bureau changed its earlier decision because a statistical direct the Bureau to (1) properly procedure used to refine the emergency and transitional shelter data proved test and evaluate procedures for to be unreliable, which lowered the quality of the data. In response, the counting people without Bureau combined the shelter data with a category called "other non- conventional housing; (2) develop institutional group quarters," a category that also includes data on people guidelines for decisions on the level of quality needed to release enumerated in several other group locations such as facilities for victims of data to the public, how to natural disasters. In the fall of 2001, the Bureau produced a heavily qualified characterize any limitations, and special report on the shelter data. A key cause of the Bureau's shifting when it is acceptable to suppress position on reporting these data appears to be its lack of well documented, data; and (3) ensure that plans for transparent, clearly defined, and consistently applied guidelines on the releasing data are clearly minimum quality necessary for releasing data. Had these guidelines been in communicated to data users. place at the time of the census, the Bureau could have been better positioned to make an objective decision on releasing these figures. The Bureau agreed with GAO’s Additionally, the Bureau could have used the guidance to explain to data recommendations, but took issue users the reasons for the decision, eliminating any appearance of censorship with our findings on the adequacy and arbitrariness. Because the Bureau did not always adequately of its data quality guidelines. communicate its plans for releasing the data, expectation gaps developed www.gao.gov/cgi-bin/getrpt?GAO-03-227. between the Bureau and entities that helped with Service-Based To view the full report, including the scope and methodology, click on the link above. Enumeration. For more information, contact Particia A. Dalton at (202) 512-6806. Contents Letter 1 Results In Brief 2 Background 4 Scope and Methodology 5 The Bureau of the Census Twice Changed Plans for Reporting Service-Based Enumeration Data 6 Census Bureau Had Few Documented Guidelines Governing the Release of Census Data 14 Conclusions 15 Recommendations for Executive Action 16 Agency Comments and Our Evaluation 17 Appendix Appendix I: Comments from the Secretary of Commerce 20 Related GAO Products 30 on the Results of the 2000 Census and Lessons Learned for a More Effective Census in 2010 Figure Figure 1: The Bureau Changed Its Original Plan to Release Emergency and Transitional Shelter Data and Combined Them with Other Noninstitutional Group Quarters Data 8 This is a work of the U.S. Government and is not subject to copyright protection in the United States. It may be reproduced and distributed in its entirety without further permission from GAO. It may contain copyrighted graphics, images or other materials. Permission from the copyright holder may be necessary should you wish to reproduce copyrighted materials separately from GAO’s product. Page i GAO-03-227 Decennial Census A United States General Accounting Office Washington, D.C. 20548 January 17, 2003 Leter The Honorable Henry A. Waxman Ranking Minority Member Committee on Government Reform House of Representatives The Honorable Danny K. Davis Ranking Minority Member Subcommittee on Civil Service, Census and Agency Organization Committee on Government Reform House of Representatives The Honorable Carolyn B. Maloney The Honorable Dennis J. Kucinich The Honorable Wm. Lacy Clay House of Representatives For the 2000 Census, the Bureau of the Census employed several initiatives to help ensure a complete and accurate count of people without conventional housing. Enumerating this segment of the population, which contains, among others, people referred to as “homeless,” has been an ongoing problem for the Bureau. In one initiative, known as Service-Based Enumeration, census enumerators attempted to count these individuals at emergency and transitional shelters, soup kitchens, regularly scheduled mobile food vans, as well as at what the Bureau calls “targeted non- sheltered outdoor locations” (TNSOL) for people living on the street at targeted locations who do not use services. To help locate and count people, the Bureau partnered with organizations providing services to the homeless and local governments, some of which put substantial resources into their efforts. In return, some of these organizations expected the Bureau to provide data that would help them plan and deliver health, employment, and other services directed toward this population. However, in its review of the emergency and transitional shelter data, the Bureau identified serious concerns with the quality of the data and concluded that the data should not be released without explanation of their extensive limitations and caveats. As a result, the Bureau decided not to separately report the emergency and transitional shelter data in the initial release of summary files as originally planned. Instead, the Bureau combined the emergency and transitional shelter data with a category Page 1 GAO-03-227 Decennial Census called “other non-institutional group quarters.” At the census tract level (small statistical subdivisions of counties) this category included soup kitchens, regularly scheduled mobile food vans, shelters for victims of domestic violence, residential care facilities providing protective oversight, staff quarters including those for nurses and interns at military and general hospitals, and living quarters for victims of natural disasters. At the block level, in addition to the categories listed above, “other non-institutional group quarters” included group homes, religious group quarters, other nonhousehold living situations, and workers’ dormitories. Aggregating the shelter numbers with these other data raised concerns among some data users that the Bureau was suppressing the results. You asked us to examine the Bureau’s decision-making process behind its change in plans. As agreed with your offices, this report examines (1) the Bureau’s plans for reporting the results of Service-Based Enumeration and its reasons for changing those plans and (2) the Bureau’s protocols for releasing data. Members of the Congress also raised concerns about the quality of Hispanic subgroup data and asked us to review the Bureau’s decision- making process for collecting and reporting ethnicity information. The results of that study are included in a companion report.1 Both reports are part of our ongoing series on the lessons learned from the 2000 Census that can help inform the planning effort for 2010. (See the Related GAO Products section for a list of reports issued to date on census issues.) Results in Brief The Bureau’s original plan for disseminating Service-Based Enumeration data was outlined in an April 1999 internal memorandum that called for the separate release of data on “emergency and transitional shelters,” but did not specify why the Bureau was not separately releasing data on the other locations enumerated during the Service-Based Enumeration—soup kitchens, regularly scheduled mobile food vans, shelters for victims of domestic violence, and targeted nonsheltered outdoor locations. The Bureau’s plan reflected its experience during the 1990 Census when it released separate counts of people found at emergency shelters, street locations, and similar locations. However, those counts proved to be incomplete. The Bureau indicated from the beginning that these 1990 1 U.S. General Accounting Office, Decennial Census: Methods for Collecting and Reporting Hispanic Subgroup Data Need Refinement, GAO-03-228 (Washington, D.C.: Jan. 17, 2003). Page 2 GAO-03-227 Decennial Census counts could not be added together to produce a count of the homeless population. Despite the Bureau’s warnings to the contrary, the data were often misinterpreted as a “homeless” count. Thus, in developing its data release plans, the Bureau took steps to ensure that the Service-Based Enumeration figures could not be added together and used as a homeless count. In January 2001, the Bureau changed its earlier decision to include the data on emergency and transitional shelters in one of its early data releases because a procedure used to refine the Service-Based Enumeration data proved to be unreliable. Although the Bureau had tested the procedure earlier in the decade, because of methodological limitations the test did not reveal any flaws. However, because problems with the procedure surfaced during the review of 2000 Census operations, the Bureau decided to combine the emergency and transitional shelter data with the “other non- institutional group quarters” category that also includes data on people enumerated in the other categories of the Service-Based Enumeration and in several other group locations, such as facilities for victims of natural disasters. As a result of this decision, the Bureau did not separately report any data from the Service-Based Enumeration in its initial release of Census 2000 data. These were the only data with separate reporting categories that the Bureau decided to collapse into another category. In the fall of 2001, the Bureau produced a special report on the emergency and transitional shelter data—including most of the same data that the Bureau earlier stated it could not release because of quality concerns. This report did not include data on targeted nonsheltered outdoor locations, or on soup kitchens and mobile food vans. The Bureau added a lengthy discussion of the limitations of the data and emphasized that they should not be interpreted as a count of the homeless population. Although the Bureau worked closely with a number of government entities, advocacy groups, and other organizations to conduct Service-Based Enumeration, reconciling its often competing data needs proved challenging. Compounding the Bureau’s difficulties, expectation gaps developed between these entities and the Bureau because the Bureau did not always clearly and consistently communicate its plans. A key cause of the Bureau’s shifting position on reporting the Service- Based Enumeration data appears to be its lack of clear, documented, and consistently applied guidelines governing the release of data from the 2000 Census. Had these guidelines been in place at the time of the census, they Page 3 GAO-03-227 Decennial Census could have helped Bureau managers decide whether to release the Service- Based Enumeration data and how to characterize these data. Additionally, the Bureau could use the guidelines to defend its decisions once they were made, thus helping to ensure that the Bureau’s decisions both are, and appear to be, completely objective. To ensure that the 2010 Census will provide data users with more complete, accurate, and useful information on people without conventional housing, we recommend that the Secretary of Commerce direct the Bureau of the Census to ensure that the procedures for enumerating and estimating segments of the population without conventional housing are properly tested and evaluated under conditions as similar to the census as possible. In addition, the Bureau should develop clearly documented, transparent, and consistently applied agencywide guidelines for releasing all census data to the public and ensure that plans for releasing data are clearly and consistently communicated to the public. The Secretary of Commerce forwarded written comments from the Bureau of the Census on a draft of this report (see app. I). The Bureau agreed with our recommendations and is taking steps to implement them, but took exception to our findings concerning the adequacy of its data quality guidelines and communication with the public. Background The procedures the Bureau used during the 1990 Census to count people without conventional housing had limitations that resulted in incomplete data.2 To address these limitations and help improve the quality of the data, the Bureau used a procedure for the 2000 Census called Service-Based Enumeration that attempted to count people where they receive services such as emergency shelters, soup kitchens, and regularly scheduled mobile food vans. Service-Based Enumeration also counted people in targeted nonsheltered outdoor locations such as encampments beneath bridges. The operation occurred from March 27 through March 29, 2000.3 2 For further information see, U.S. General Accounting Office, 1990 Census: Limitations in Methods and Procedures to Include the Homeless, GAO/GGD-92-1 (Washington, D.C.: Dec. 30, 1991). 3 For information on the conduct of Service-Based Enumeration see, U.S. General Accounting Office, 2000 Census: Progress Report on the Mail Response Rate and Key Operations, GAO/T-GGD/AIMD-00-136 (Washington, D.C.: Apr. 5, 2000). Page 4 GAO-03-227 Decennial Census According to Bureau officials, Service-Based Enumeration was not designed, and was never intended, to provide a specific count of homeless persons. Instead, the operation was part of a larger effort to count people without conventional housing, including people in “institutional group quarters” such as correctional facilities, nursing homes, and mental hospitals, and “non-institutional group quarters” such as college dormitories, military quarters, and group homes. Service-Based Enumeration counted people in specific categories of noninstitutional group quarters. To help ensure a complete count of people without conventional housing, the Bureau partnered with local governments and community advocacy groups to obtain lists of service locations and to assist with the enumeration.4 In some cases, the Bureau hired clients of the advocacy groups and other people trusted by the homeless to conduct Service-Based Enumeration. For example in Atlanta, an advocacy group for homeless veterans helped the Bureau employ homeless veterans to improve the count of this population. Local governments helped the Bureau as well, often investing considerable resources. For example, Los Angeles paid to keep its city-run shelters open on the night they were enumerated so that people using their services could be counted. Scope and To address your concerns about the Bureau’s dissemination of data on persons without conventional housing, we agreed to examine (1) the Methodology Bureau’s plans for reporting the results of Service-Based Enumeration and its reasons for changing those plans and (2) the Bureau’s protocols for releasing data. To accomplish these objectives, we interviewed key Bureau officials and reviewed relevant Bureau documents and data such as operational plans, decision memorandums, and the Bureau’s partnership program evaluation. In order to obtain the perspective of data users, partners, and stakeholders, we conducted in-person and telephone interviews with homeless advocates, local government officials, and representatives of public service agencies in New York City, Los Angeles, Cleveland, Atlanta, and Washington, D.C. These cities had large numbers of people without 4 For additional information see, U.S. General Accounting Office, 2000 Census: Review of Partnership Program Highlights Best Practices for Future Operations, GAO-01-579 (Washington, D.C.: Aug. 20, 2001). Page 5 GAO-03-227 Decennial Census conventional housing and they were actively involved with the Bureau during the 2000 Census. The organizations we contacted also provided relevant documentation, such as comprehensive file documents relating to partnership activities. In addition to the above locations, we did our audit work at Bureau headquarters in Suitland, Maryland. Our audit work was conducted from April 2002 through September 2002 in accordance with generally accepted government auditing standards. We requested comments on a draft of this report from the Secretary of Commerce. On November 21, 2002, the Secretary forwarded the Bureau’s written comments on the draft (see app. I). We address these comments at the end of this report. The Bureau of the Under the Bureau’s original plan for releasing Service-Based Enumeration data in Summary File-1 (SF-1), 5 the emergency and transitional shelter Census Twice Changed count was one of several categories of noninstitutional group quarters data Plans for Reporting that were to be reported separately. Other people counted in the Service- Based Enumeration, including people counted at targeted nonsheltered Service-Based outdoor locations, soup kitchens, and regularly scheduled mobile food Enumeration Data vans, were to be combined and reported under the category “other non- institutional group quarters.” This category also included residential care facilities providing protective oversight, shelters against domestic violence, staff dormitories for nurses and interns at military and general hospitals, and living quarters for victims of natural disasters. 5 The SF-1 is the summary file in which the Bureau presents population and housing data for the total population. Other than the Census 2000 Redistricting Data Summary File, the SF-1 file is the first product released after the census. Page 6 GAO-03-227 Decennial Census This decision was documented in an April 1999 internal memorandum from the Bureau’s Assistant Division Chief for Special Population Statistics to the Assistant Division Chief for Census Programs. The Service-Based Enumeration operation took place a year later, in March 2000. The April 1999 plan was in large part a reaction to the challenges the Bureau faced counting the emergency shelter and street population during the 1990 Census. Although the Bureau disseminated separate counts of people found at emergency shelters, preidentified street locations, and similar sites, the counts proved to be incomplete. 6 Moreover, the Bureau stated in its October 2001 report that despite its warnings to the contrary, the data were sometimes misinterpreted as a “homeless” count. The October report does not offer an example of this, but the misinterpretation clearly played a role in a lawsuit against the Bureau. 7 As a result, when designing the 2000 census, the Bureau attempted to both improve the count and take precautions to ensure that the Service-Based Enumeration count would not be misconstrued as a count of the homeless. The Bureau’s data dissemination plans took into account the recommendations of the Commerce Secretary’s 2000 Census Advisory Committee, a panel that included representatives of advocacy and other groups (including representatives from organizations that represent local governments) that met periodically to review the Bureau’s plans. The homeless population was represented by the National Coalition for the Homeless—an advocacy group that coordinates a network of 300 state and local housing and homeless organizations. In its January 1999 final report, the Census 2000 Advisory Committee recommended that special attention be paid to tabulating the results of Service-Based Enumeration and targeted outdoor enumerations so that they could not be aggregated and used as a homeless count. 6 For more information on the 1990 count see, U.S. General Accounting Office, Counting the Homeless: Limitations of 1990 Census Results and Methodology, GAO/T-GGD-91-29 (Washington, D.C.: May 9, 1991) and 1990 Census: Limitations in Methods and Procedures to Include the Homeless, GAO/GGD-92-1 (Washington, D.C.: Dec. 30, 1991). 7 One example was the National Law Center On Homelessness and Poverty, et. al., v. Brown, where plaintiffs alleged that the Department of Commerce effectively excluded the nation’s homeless population from the 1990 decennial census in violation of the Constitution and the Administrative Procedure Act. The District Court denied the plaintiff’s claim (1994 WL 521334) (D.D.C. Sept. 15, 1994) and the Court of Appeals dismissed the lawsuit for lack of standing. 91 F.3d 178 (D.C. Cir. 1996). Page 7 GAO-03-227 Decennial Census The Bureau Changed Its In January 2001, 5 months before the SF-1 release, the Bureau reversed its Dissemination Plans April 1999 decision to release emergency and transitional shelter data separately because of “data quality concerns.” Instead, as shown in figure Because of Data Quality 1, the Bureau planned to combine the emergency and transitional Concerns Figure 1: The Bureau Changed Its Original Plan to Release Emergency and Transitional Shelter Data and Combined Them with Other Noninstitutional Group Quarters Data Original plan (April 1999) Revised plan (January 2001) Group quarters Group quarters Institutionalized population Non-Institutionalized NoninstitutionalizedPopulation population reporting categories reporting categories ■ Correctional institutions ■ College dorms Noninstitutionalized population ■ Nursing homes reporting categories ■ Military quarters ■ Hospitals ■ Group homes ■ College dorms ■ Juvenile institutions ■ Religious group quarters ■ Military quarters ■ Worker dormitories ■ Group homes ■ Crews of maritime vessels ■ Religious group quarters ■ Worker dormitories ■ Emergency and transitional shelters ■ Crews of maritime vessels ■ Other noninstitutional group quarters ■ Other noninstitutional group quarters The other noninstitutional group quarters categories are: Soup kitchens and regularly scheduled food vans The other noninstitutional group quarters categories are: Targeted nonsheltered outdoor locations Emergency and transitional shelters Residential facilities "providing protective oversight" Living quarters for victims of natural disasters Soup kitchens and regularly scheduled food vans Staff residents of institutions Targeted nonsheltered outdoor locations Shelters for victims of domestic violence Residential facilities "providing protective oversight" Living quarters for victims of natural disasters Staff residents of institutions Shelters for victims of domestic violence Source: GAO depiction based on Bureau of the Census data. Page 8 GAO-03-227 Decennial Census shelter data with the “other non-institutional group quarters.” This category contained data on a variety of living arrangements including facilities for natural disaster victims. The Bureau’s decision was contained in an internal Bureau memorandum from the Chief of the Population Division to the Chief of the Decennial Systems and Contracts Management Office. Bureau officials told us that the decision to exclude a separate emergency and transitional shelter count in SF-1 was made between December 2000 and January 2001, by the Director of the Decennial Census with input from the Associate Director Decennial Census, the Population Division, the Associate Director for Demographic Programs, the Decennial Management Division, and the Decennial Statistical Studies Division.8 According to Bureau officials, their concerns focused on the accuracy of a new statistical procedure called “multiplicity estimation” that adjusted the number counted to better reflect the number of actual shelter users. Because Service-Based Enumeration only counted people who were at these facilities on the day of enumeration, the Bureau intended to use multiplicity estimation to calculate the number of people who used these facilities but were not present during Service-Based Enumeration. The multiplicity estimation procedure was based on information from those who were counted and on the number of times they used the service facilities in the prior week. An estimate of people not counted on the day of enumeration was added to the count of people who were. According to the Bureau, the multiplicity estimates tested well during the 1998 dress rehearsal for the 2000 Census possibly because the three rehearsal sites did not offer large enough sample sizes of the appropriate populations to adequately test this procedure.9 However, during the 2000 Census the Bureau found that a census question pertaining to facility usage upon which the multiplicity estimates were based had a low response rate. Moreover, the Bureau found that respondents, particularly in shelters, did not answer the question accurately. Due to data quality concerns, the Bureau decided not to use multiplicity estimation to adjust the data and consequently decided not to report the data separately. 8 Although the Bureau changed its plan in January 2001, the technical documentation for SF-1, released at the same time, still indicated that emergency and transitional shelter data would be separately reported. 9 The dress rehearsal for the 2000 Census was conducted in Sacramento, California, City of Columbia, South Carolina, and Menominee County, Wisconsin, including the Menominee Indian Reservation. The dress rehearsal was designed to test the overall design of the 2000 Census. Page 9 GAO-03-227 Decennial Census Bureau officials said they did not announce the change in plans because they were still evaluating the problems with the data. It was not until June 2000 that the Bureau began recalculating the data and making a final decision on which categories to aggregate. Ultimately, the Bureau did not report any of the Service-Based Enumeration data separately in SF-1. Emergency and transitional shelter data were the only data that were to be released in SF-1 under separate reporting categories that the Bureau decided to combine with another category. The Bureau Produced a The release of the SF-1 data in June 2001 produced public discussion in the Special Report in October press, among census partners, and in the Congress about the Bureau’s decision to not separately release Service-Based Enumeration data. In a 2001 on the Emergency and briefing for staff of the House Committee on Governmental Affairs, the Transitional Shelter Associate Director of the Decennial Census announced that the Bureau Population planned to produce a separate report on the emergency and transitional shelter data. In October 2001, the Bureau issued a special report, entitled Emergency and Transitional Shelter Population: 2000. This report separately identified emergency and transitional shelter data for various levels of geography down to the census tract level with 100 or more people in emergency and transitional shelters. The report did not include data for the populations in targeted nonsheltered outdoor locations, soup kitchens, regularly scheduled mobile food vans, and shelters for domestic violence. The 17-page report contains an extensive discussion on the limitations of the data. For example, the Bureau noted that the data in the report should not be construed as a count of people without conventional housing. Moreover, the emergency and transitional shelter data at the census tract level are not in the hard copy, but rather in the Internet version of the report.10 The Bureau stated that all Census 2000 data at the tract level are available on the Internet and are not available in printed reports. The October report contains most of the same data that were to be released under the April 1999 dissemination plan for SF-1. The Bureau asserted that the data quality concerns with the emergency and transitional shelter data (cited when it changed the plan to release these data in SF-1) required that the data be presented in a manner that allowed the Bureau to clearly 10 U.S. Bureau of the Census, Population in Emergency and Transitional Shelters (PHC-T-12) (Washington, D.C.: Oct. 30, 2001). http://www.census.gov/population/www/cen2000/phc-t12.html Page 10 GAO-03-227 Decennial Census outline the data’s limitations. The October 2001 report contained an extended discussion of these limitations. The October 2001 report also identified reasons the Bureau did not (and never planned to) separately release data on people counted at targeted nonsheltered outdoor locations, soup kitchens, regularly scheduled mobile food vans, and shelters for victims of domestic violence, including the following. • People without conventional housing who were at outside locations other than the targeted nonsheltered outdoor locations identified for the census were not included in the TNSOL operation. • For the purposes of the TNSOL operation, the definition of “outdoor” excluded both mobile and transient locations used by people experiencing homelessness as well as abandoned buildings. • The option was given to the individuals found at soup kitchens and regularly scheduled mobile food vans to select “usual home elsewhere.” For example, if an individual enumerated at a soup kitchen listed a usual home elsewhere, then that person was tabulated at their usual residence and not at the service location. Therefore, the data on this population would not reflect a true count of the individuals using these services. Prior to publicly releasing the October report, the Bureau asked two representatives from the National Coalition for the Homeless to review a draft of the portion of the report that described the limitations of the data. The National Coalition for the Homeless commented extensively on the section containing the caveats and limitations in order to strengthen the report. A member of the Board of Directors for the National Coalition for the Homeless told us that he provided this feedback both as an academician and a stakeholder. Bureau officials stated that because of its position on the Bureau’s Census Advisory Committee, the National Coalition for the Homeless was the only advocacy group that reviewed any portion of the October 2001 report prior to its publication. Meeting Data Users’ Needs The controversy surrounding the release of the combined Service-Based Proved Challenging Enumeration data highlights the challenges the Bureau faced in 2000 trying to meet the needs of various data users and the work the Bureau still needs to do when planning for the 2010 Census to better reconcile those needs. For example, several organizations we contacted favored the separate Page 11 GAO-03-227 Decennial Census release of the Service-Based Enumeration data categories. Indeed, local government officials we talked to in New York City believed that the data would help with grant applications, projections about future service needs, and determining their success in getting people off the streets and into shelters. The Executive Director of the Northeast Ohio Coalition for the Homeless stated that the city of Cleveland does not do its own count of this population and, therefore, the Bureau numbers are the only ones available on this segment of the population. Los Angeles city officials wanted the Service-Based Enumeration data so they could better target their services and, like Cleveland, Los Angeles did not have its own data. Several of these entities stated that the potential misuse of data was not a valid reason for not separately releasing data. In addition, the majority of the organizations we contacted partnered with the Bureau anticipating that they would be able to use the Service-Based Enumeration data to evaluate whether improvements were made in enumerating local populations without conventional housing in 2000 compared to 1990. The Assistant City Attorney of Los Angeles estimated that Los Angeles spent about $300,000 on the effort to improve the count of Los Angeles’s people without conventional housing. For example, as part of an extensive effort to help the Bureau develop a list of targeted nonsheltered outdoor locations, the city provided senior Bureau staff with a helicopter tour over some outdoor locations where people without conventional housing lived. The Assistant City Attorney of Los Angeles stated that she believed the city would get the targeted nonsheltered outdoor locations data that they helped collect and wanted to review. In addition, because of the Bureau’s focus on counting people at shelters, the city kept shelters open on the night of the enumeration at its own expense even though shelters in Los Angeles typically do not have many people during warm weather. Los Angeles expected to have detailed data to use to evaluate the effectiveness of its resource allocation. However, the National Coalition for the Homeless and other advocates of the homeless opposed the separate release of any of the Service-Based Enumeration data. They were concerned that these data could be misused as a count of the homeless population and lead to flawed decision-making by policymakers. Ultimately, the Bureau left a number of data users unsatisfied. Those who wanted the Service-Based Enumeration categories released separately did not feel the Bureau met their expectations with the data released in SF-1 or with the release of the October report. Users who opposed the separate Page 12 GAO-03-227 Decennial Census release of the data and were pleased that SF-1 combined the Service-Based Enumeration components with other data were displeased that the October 2001 report was released. The difficulties the Bureau experienced trying to reconcile the competing needs and interests of data users illustrates the importance of effective communication between the Bureau and its key data users and partners to ensure no expectation gaps develop. More than just a good business practice, federal internal control standards require agencies to have effective external communications with groups that can have a serious impact on programs, projects, operations, and other activities.11 However, our conversations with several Bureau partners and our review of Bureau documents suggest that communications were sometimes vague and insufficient. For example, although the April 1999 memorandum that outlined the Bureau’s initial data dissemination plans was written a year before the 2000 Census, this information may not have been effectively communicated to the Bureau’s partners. Indeed, at a Capitol Hill briefing on this topic in June 2001, Bureau officials themselves acknowledged that they did not do a good job of communicating on this issue. Some of the partners we spoke to indicated that had they known earlier about the Bureau’s plans to limit the release of Service-Based Enumeration data they might have focused their resources on different census operations. Further, our review of Bureau documents indicated that the information on the “official plan” for the release of the different Service-Based Enumeration categories of data was limited and inconsistent. Some partners stated that they did not know that the Bureau never intended to report the targeted nonsheltered outdoor location data. Although the Bureau made numerous presentations on Service-Based Enumeration that emphasized there would be no count of the homeless, the Bureau provided little detail on how components of Service-Based Enumeration would actually be presented. In the absence of clear communication from the Bureau, partners developed their own expectations of what would be released. Several of the local officials and advocates that we spoke to expected that the data would be released in the 11 U.S. General Accounting Office, Internal Control Standards, Internal Control Management and Evaluation Tool (August 2001) and U.S. General Accounting Office, Internal Control Standards for Internal Control in the Federal Government (November 1999). Page 13 GAO-03-227 Decennial Census same detail as it was in 1990, because they were not told otherwise. For example, a Los Angeles government official said that the Bureau stated it would not provide a homeless count in 1990, but it still released the street count data separately. By focusing resources on counting specific categories of the population, the Bureau may have created expectations that there would be a count of that population. Census Bureau Had A cause of the Bureau’s shifting position on reporting the components of Service-Based Enumeration appears to be its lack of documented, clear, Few Documented transparent, and consistently applied guidelines governing the release of Guidelines Governing data from the 2000 Census. Except for some guidance aimed at protecting the confidentiality of census records, the Bureau had few written the Release of Census guidelines on the level of quality needed to release data to the public. Had Data these guidelines been in place during the decennial census, they could have informed the Bureau’s decision on whether to release the Service-Based Enumeration data, how to characterize these data, and help defend the decision after it was made. Such guidelines could also provide a basis ahead of time for expectations about the conditions under which data will or will not be released. Although Bureau officials emphasized that the Bureau has a long tradition of high standards and procedures that yield quality data (to its credit, the Bureau’s quality assurance practices identified the problem with the multiplicity estimator), the officials acknowledged that these standards were primarily part of the agency’s institutional knowledge. The written guidance that did exist appeared to be vague and insufficient for making consistent decisions on the quality thresholds needed for releasing data to the public, and the circumstances under which it might be appropriate to suppress certain data. Page 14 GAO-03-227 Decennial Census According to the Bureau’s Associate Director for Methodology and Standards, a technical paper issued in 1974 and revised in 1987 contained the Bureau’s only written guidelines for discussion and presentation of errors in data. This paper noted that, “[e]stimates for individual cells of a published table should not be suppressed solely because they are subject to large sampling errors or large nonsampling variances, provided users are given adequate caution of the lack of reliability of the data. On the other hand, data known to have very serious bias may be suppressed.”12 A Newly Created Bureau In 2000, the Bureau initiated a new quality assurance program to document Program Could Provide Bureau-wide protocols designed to ensure the quality of data collected and disseminated by the Bureau. The Bureau’s Methodology and Standards Guidelines for Releasing Council is charged with setting statistical and survey quality standards and Data guidelines for Bureau surveys and censuses. In support of this role, the council has established a quality framework in which the demographic, economic, and decennial areas can share and support common principles, standards, and guidelines. The quality framework covers eight unique areas, one of which is dissemination. Because this Bureau program is in its initial stages, we could not evaluate it. However, Bureau officials believe that the program is a significant first step in addressing the lack of agencywide written guidelines for releasing data. The initiative appears to be consistent with Office of Management and Budget guidelines issued in February 2002 requiring federal agencies to issue their own guidance for ensuring and maximizing the quality, objectivity, utility, and integrity of information disseminated by the agency.13 As the Bureau develops its guidelines, it will be important that they be well documented, transparent, clearly defined, and consistently applied. Conclusions Although Service-Based Enumeration was designed to address the challenges the Bureau encountered during the 1990 Census in obtaining a complete count of people without conventional housing, the Bureau’s experience during the 2000 Census suggests that tallying this population 12 U.S. Department of Commerce, Technical Paper 32, Standards for Discussion and Presentation of Errors in Data (March 1974), p. 3. 13 Issued at 67 Fed. Reg. 8452 (Feb. 22, 2002). Page 15 GAO-03-227 Decennial Census group remains problematic. Moreover, the Bureau’s difficulties were compounded by its shifting position on how to report the data once they were collected. A number of government, community, and advocacy organizations helped the Bureau enumerate this population group. However, the Bureau, by first planning to release the data one way, then changing the decision, and ultimately releasing the data anyway—all for reasons that were not clearly articulated to the Bureau’s stakeholders— raised questions about the Bureau’s decision-making on data quality issues. As noted at the beginning of this report, related questions have also been raised about how the Bureau collected and reported data on Hispanic subgroups. To the extent similar incidents occur in the future, they could undermine public confidence in the accuracy and credibility of Bureau data. Thus, as the Bureau plans for the 2010 Census, it will be important for it to refine its methods for enumerating people living in unconventional housing and reporting the resulting data, in part by properly testing and evaluating those methods. As noted earlier, the Bureau could not properly test a key statistical technique during the census dress rehearsal because the sample size was too small. Moreover, while addressing the competing needs and desires of data users will likely remain a considerable challenge, it will be important for the Bureau to more effectively articulate its plans to avoid the expectation gaps that occurred during 2000. The Bureau’s plans for collecting data on persons without conventional housing need to specify how the Bureau plans to separately report these data. Bureau-wide guidelines on the level of quality needed to release data to the public, on how and when to document data limitations, and on the circumstances under which it is acceptable to suppress data, could help the Bureau be more accountable and consistent in its dealings with data users and stakeholders, and help ensure that the Bureau’s decisions both are, and appear to be, totally objective. Recommendations for To ensure that the 2010 Census will provide public data users with more complete, accurate, and useful information on the segment of the Executive Action population without conventional housing, we recommend that the Secretary of Commerce direct the Director of the Bureau of the Census to do the following. Page 16 GAO-03-227 Decennial Census 1. Ensure that all procedures for enumerating and estimating segments of the population without conventional housing are properly tested and evaluated under conditions as similar to the census as possible. 2. Develop agencywide guidelines for Bureau decisions on the level of quality needed to release data to the public, how to characterize any limitations in the data, and when it is acceptable to suppress the data for reasons other than protecting the confidentiality of respondents. Ensure that these guidelines are documented, transparent, clearly defined, and consistently applied. 3. Ensure that the Bureau’s plans for releasing data are clearly and consistently communicated with the public. Agency Comments and The Secretary of Commerce forwarded written comments from the Bureau of the Census on a draft of this report (see app. I). The Bureau agreed with Our Evaluation each of our recommendations and, as indicated in the letter, is taking steps to implement them. However, it expressed several general concerns about our findings. The Bureau’s principal concerns and our response are presented below. The Bureau also suggested minor wording changes to provide additional context and clarification. We accepted the Bureau’s suggestions and made changes to the text as appropriate. The Bureau took exception to our findings concerning the adequacy of its data quality guidelines, noting that the Bureau’s decisions regarding the release and characterization of emergency and transitional shelter data were based on established guidelines for data quality. However, the Bureau did not cite any written guidelines to support its position. As noted in our report, Bureau officials, including the Associate Director for Methodology and Standards, told us that the Bureau had few written guidelines, standards, or procedures related to the quality of data released to the public. In this report we acknowledge the Bureau’s tradition of high standards and procedures that yield quality data. However, according to the Bureau, these standards are generally undocumented and part of the agency’s institutional knowledge. To provide a basis for consistent decision-making and clear communication within the Bureau and to the public, guidelines on the quality of data released to the public must be fully documented, transparent, clearly defined, and consistently applied. Additionally, the Bureau said that when data do not meet an acceptable level of quality, it considers various options for modifying its dissemination Page 17 GAO-03-227 Decennial Census plans. The Bureau’s decision to delay the release of the emergency and transitional shelter data may have been entirely appropriate. Our concern is not that the Bureau changed its plans, but that it could not provide us its guidelines for determining an acceptable level of quality or clearly indicate how it determined that the data did not meet minimal quality standards for release in SF-1. The Bureau also commented that its decisions regarding the distribution of data from SF-1 were well publicized and that the only change in Bureau plans for the release of Service-Based Enumeration data was the decision to delay release of the emergency and transitional shelter data. This report focused on the changing plans for the release of the emergency and transitional shelter data and noted that the Bureau never intended to release any other data from the Service-Based Enumeration. However, we found that the Bureau did not effectively communicate its decisions with its partners or the public. Decisions on the release of the emergency and transitional shelter data were contained in internal decision memoranda. We found that these decisions were not always reflected in new releases of the SF-1 documentation. Although Bureau officials told us that they always intended to produce a separate report on emergency and transitional shelter data, they did not make this intention public when the SF-1 data were released. Some stakeholders did not realize that the Bureau was not releasing emergency and transitional shelter data with SF-1 until they examined the SF-1 data. As we stated in our report, these communication problems can undermine stakeholder and public confidence in the Bureau and its products. As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will send copies of this report to the Chairman of the House Committee on Government Reform, the Chairman of the Subcommittee on Civil Service, Census and Agency Organization, the Secretary of Commerce, and the Director of the Bureau of the Census. Copies will be made available to others on request. This report will also be available at no charge on GAO’s home page at http://www.gao.gov. Page 18 GAO-03-227 Decennial Census Please contact me on (202) 512-6806 or by e-mail at email@example.com if you have any questions. Other key contributors to this report were Robert Goldenkoff, Timothy Wexler, Elizabeth Powell, Chris Miller, James Whitcomb, Ty Mitchell, Robert Parker, and Michael Volpe. Patricia A. Dalton Director Strategic Issues Page 19 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce AA ppp ep ned n x id e x Iis Page 20 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 21 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 22 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 23 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 24 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 25 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 26 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 27 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 28 GAO-03-227 Decennial Census Appendix I Comments from the Secretary of Commerce Page 29 GAO-03-227 Decennial Census Related GAO Products on the Results of the 2000 Census and Lessons Learned for a More Effective Census in 2010 2000 Census: Refinements to Full Count Review Program Could Improve Future Data Quality. GAO-02-562. Washington, D.C.: July 3, 2002. 2000 Census: Coverage Evaluation Matching Implemented As Planned, but Census Bureau Should Evaluate Lessons Learned. GAO-02-297. Washington, D.C.: March 14, 2002. 2000 Census: Best Practices and Lessons Learned for a More Cost- Effective Nonresponse Follow-Up. GAO-02-196. Washington, D.C.: February 11, 2002. 2000 Census: Coverage Evaluation Interviewing Overcame Challenges, but Further Research Needed. GAO-02-26. Washington, D.C.: December 31, 2001. 2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control Weaknesses at the U.S. Census Bureau. GAO-02-30. Washington, D.C.: December 28, 2001. 2000 Census: Significant Increase in Cost Per Housing Unit Compared to 1990 Census. GAO-02-31. Washington, D.C.: December 11, 2001. 2000 Census: Better Productivity Data Needed for Future Planning and Budgeting. GAO-02-4. Washington, D.C.: October 4, 2001. 2000 Census: Review of Partnership Program Highlights Best Practices for Future Operations. GAO-01-579. Washington, D.C.: August 20, 2001. Decennial Censuses: Historical Data on Enumerator Productivity Are Limited. GAO-01-208R. Washington, D.C.: January 5, 2001. 2000 Census: Information on Short- and Long-Form Response Rates. GAO/GGD-00-127R. Washington, D.C.: June 7, 2000. (450102) Page 30 GAO-03-227 Decennial Census GAO’s Mission The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. Obtaining Copies of The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- GAO Reports and text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents Testimony using key words and phrases. You can print these documents in their entirety, including charts and other graphics. Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO Mailing Lists” under “Order GAO Products” heading. Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check or money order should be made out to the Superintendent of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single address are discounted 25 percent. Orders should be sent to: U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C. 20548 To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202) 512-6061 To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm Waste, and Abuse in E-mail: firstname.lastname@example.org Federal Programs Automated answering system: (800) 424-5454 or (202) 512-7470 Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C. 20548 United States Presorted Standard General Accounting Office Postage & Fees Paid Washington, D.C. 20548-0001 GAO Permit No. GI00 Official Business Penalty for Private Use $300 Address Service Requested
Decennial Census: Methods for Collecting and Reporting Data on the Homeless and Others without Conventional Housing Need Refinement
Published by the Government Accountability Office on 2003-01-17.
Below is a raw (and likely hideous) rendition of the original report. (PDF)