oversight

Decennial Census: Methods for Collecting and Reporting Data on the Homeless and Others without Conventional Housing Need Refinement

Published by the Government Accountability Office on 2003-01-17.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States General Accounting Office

GAO            Report to Congressional Requesters




January 2003
               DECENNIAL CENSUS
               Methods for Collecting
               and Reporting Data on
               the Homeless and
               Others without
               Conventional Housing
               Need Refinement




GAO-03-227
               a
                                               January 2003


                                               DECENNIAL CENSUS

                                               Methods for Collecting and Reporting
Highlights of GAO-03-227, a report to          Data on the Homeless and Others
Congressional Requesters
                                               Without Conventional Housing Need
                                               Refinement


The Bureau of the Census                       The Bureau's original plan for releasing Service-Based Enumeration data
partnered with local governments,              was outlined in an April 1999 internal memorandum that called for the
advocacy groups, and other                     separate release of data on people counted at "emergency and transitional
organizations to help it enumerate             shelters." The Bureau planned to combine other components of Service-
people without conventional                    Based Enumeration, including people counted at soup kitchens, regularly
housing. Counting this
population—which includes shelter
                                               scheduled mobile food vans, and certain outdoor locations, into a single
residents and the homeless—has                 category. Driving the Bureau's decision was its experience during the 1990
been a longstanding challenge for              Census when it released separate counts of people found at shelters, on the
the Bureau. A number of                        street, and similar locations that proved to be incomplete. The Bureau also
organizations put substantial                  tried to ensure that the Service-Based Enumeration figures could not be used
resources into an operation the                as a "homeless" count, because it was not designed to provide a specific
Bureau called Service-Based                    count of the homeless. Instead, the operation was part of a larger effort to
Enumeration. In return, some                   count people without conventional housing.
expected the Bureau to provide
data that would help them plan and
                                               The Homeless Are Hard to Enumerate
deliver employment, health, and
other services. However, the
Bureau did not release the data as
planned, which raised questions
about the Bureau’s decision-
making on data quality issues. In
response to a congressional
request, GAO examined the
Bureau’s decision-making process
behind its change in plans.

                                               Source: U.S. Bureau of the Census.

The Secretary of Commerce should               In January 2001, the Bureau changed its earlier decision because a statistical
direct the Bureau to (1) properly              procedure used to refine the emergency and transitional shelter data proved
test and evaluate procedures for               to be unreliable, which lowered the quality of the data. In response, the
counting people without                        Bureau combined the shelter data with a category called "other non-
conventional housing; (2) develop
                                               institutional group quarters," a category that also includes data on people
guidelines for decisions on the
level of quality needed to release             enumerated in several other group locations such as facilities for victims of
data to the public, how to                     natural disasters. In the fall of 2001, the Bureau produced a heavily qualified
characterize any limitations, and              special report on the shelter data. A key cause of the Bureau's shifting
when it is acceptable to suppress              position on reporting these data appears to be its lack of well documented,
data; and (3) ensure that plans for            transparent, clearly defined, and consistently applied guidelines on the
releasing data are clearly                     minimum quality necessary for releasing data. Had these guidelines been in
communicated to data users.                    place at the time of the census, the Bureau could have been better
                                               positioned to make an objective decision on releasing these figures.
The Bureau agreed with GAO’s                   Additionally, the Bureau could have used the guidance to explain to data
recommendations, but took issue                users the reasons for the decision, eliminating any appearance of censorship
with our findings on the adequacy
                                               and arbitrariness. Because the Bureau did not always adequately
of its data quality guidelines.
                                               communicate its plans for releasing the data, expectation gaps developed
www.gao.gov/cgi-bin/getrpt?GAO-03-227.         between the Bureau and entities that helped with Service-Based
To view the full report, including the scope
and methodology, click on the link above.
                                               Enumeration.
For more information, contact Particia A.
Dalton at (202) 512-6806.
Contents



Letter                                                                                                                 1
                            Results In Brief                                                                           2
                            Background                                                                                 4
                            Scope and Methodology                                                                      5
                            The Bureau of the Census Twice Changed Plans for Reporting
                              Service-Based Enumeration Data                                                           6
                            Census Bureau Had Few Documented Guidelines Governing the
                              Release of Census Data                                                                   14
                            Conclusions                                                                                15
                            Recommendations for Executive Action                                                       16
                            Agency Comments and Our Evaluation                                                         17


Appendix
              Appendix I:   Comments from the Secretary of Commerce                                                    20


Related GAO Products                                                                                                   30
on the Results of the
2000 Census and
Lessons Learned for a
More Effective Census
in 2010

Figure                      Figure 1: The Bureau Changed Its Original Plan to Release
                                      Emergency and Transitional Shelter Data and Combined
                                      Them with Other Noninstitutional Group Quarters Data                             8




                             This is a work of the U.S. Government and is not subject to copyright protection in the
                             United States. It may be reproduced and distributed in its entirety without further
                             permission from GAO. It may contain copyrighted graphics, images or other materials.
                             Permission from the copyright holder may be necessary should you wish to reproduce
                             copyrighted materials separately from GAO’s product.




                            Page i                                                       GAO-03-227 Decennial Census
A
United States General Accounting Office
Washington, D.C. 20548



                                    January 17, 2003                                                               Leter




                                    The Honorable Henry A. Waxman
                                    Ranking Minority Member
                                    Committee on Government Reform
                                    House of Representatives

                                    The Honorable Danny K. Davis
                                    Ranking Minority Member
                                    Subcommittee on Civil Service,
                                      Census and Agency Organization
                                    Committee on Government Reform
                                    House of Representatives

                                    The Honorable Carolyn B. Maloney
                                    The Honorable Dennis J. Kucinich
                                    The Honorable Wm. Lacy Clay
                                    House of Representatives

                                    For the 2000 Census, the Bureau of the Census employed several initiatives
                                    to help ensure a complete and accurate count of people without
                                    conventional housing. Enumerating this segment of the population, which
                                    contains, among others, people referred to as “homeless,” has been an
                                    ongoing problem for the Bureau. In one initiative, known as Service-Based
                                    Enumeration, census enumerators attempted to count these individuals at
                                    emergency and transitional shelters, soup kitchens, regularly scheduled
                                    mobile food vans, as well as at what the Bureau calls “targeted non-
                                    sheltered outdoor locations” (TNSOL) for people living on the street at
                                    targeted locations who do not use services. To help locate and count
                                    people, the Bureau partnered with organizations providing services to the
                                    homeless and local governments, some of which put substantial resources
                                    into their efforts. In return, some of these organizations expected the
                                    Bureau to provide data that would help them plan and deliver health,
                                    employment, and other services directed toward this population.

                                    However, in its review of the emergency and transitional shelter data, the
                                    Bureau identified serious concerns with the quality of the data and
                                    concluded that the data should not be released without explanation of their
                                    extensive limitations and caveats. As a result, the Bureau decided not to
                                    separately report the emergency and transitional shelter data in the initial
                                    release of summary files as originally planned. Instead, the Bureau
                                    combined the emergency and transitional shelter data with a category



                                    Page 1                                             GAO-03-227 Decennial Census
                   called “other non-institutional group quarters.” At the census tract level
                   (small statistical subdivisions of counties) this category included soup
                   kitchens, regularly scheduled mobile food vans, shelters for victims of
                   domestic violence, residential care facilities providing protective oversight,
                   staff quarters including those for nurses and interns at military and general
                   hospitals, and living quarters for victims of natural disasters. At the block
                   level, in addition to the categories listed above, “other non-institutional
                   group quarters” included group homes, religious group quarters, other
                   nonhousehold living situations, and workers’ dormitories. Aggregating the
                   shelter numbers with these other data raised concerns among some data
                   users that the Bureau was suppressing the results.

                   You asked us to examine the Bureau’s decision-making process behind its
                   change in plans. As agreed with your offices, this report examines (1) the
                   Bureau’s plans for reporting the results of Service-Based Enumeration and
                   its reasons for changing those plans and (2) the Bureau’s protocols for
                   releasing data.

                   Members of the Congress also raised concerns about the quality of
                   Hispanic subgroup data and asked us to review the Bureau’s decision-
                   making process for collecting and reporting ethnicity information. The
                   results of that study are included in a companion report.1 Both reports are
                   part of our ongoing series on the lessons learned from the 2000 Census that
                   can help inform the planning effort for 2010. (See the Related GAO
                   Products section for a list of reports issued to date on census issues.)



Results in Brief   The Bureau’s original plan for disseminating Service-Based Enumeration
                   data was outlined in an April 1999 internal memorandum that called for the
                   separate release of data on “emergency and transitional shelters,” but did
                   not specify why the Bureau was not separately releasing data on the other
                   locations enumerated during the Service-Based Enumeration—soup
                   kitchens, regularly scheduled mobile food vans, shelters for victims of
                   domestic violence, and targeted nonsheltered outdoor locations. The
                   Bureau’s plan reflected its experience during the 1990 Census when it
                   released separate counts of people found at emergency shelters, street
                   locations, and similar locations. However, those counts proved to be
                   incomplete. The Bureau indicated from the beginning that these 1990

                   1
                     U.S. General Accounting Office, Decennial Census: Methods for Collecting and Reporting
                   Hispanic Subgroup Data Need Refinement, GAO-03-228 (Washington, D.C.: Jan. 17, 2003).




                   Page 2                                                     GAO-03-227 Decennial Census
counts could not be added together to produce a count of the homeless
population. Despite the Bureau’s warnings to the contrary, the data were
often misinterpreted as a “homeless” count. Thus, in developing its data
release plans, the Bureau took steps to ensure that the Service-Based
Enumeration figures could not be added together and used as a homeless
count.

In January 2001, the Bureau changed its earlier decision to include the data
on emergency and transitional shelters in one of its early data releases
because a procedure used to refine the Service-Based Enumeration data
proved to be unreliable. Although the Bureau had tested the procedure
earlier in the decade, because of methodological limitations the test did not
reveal any flaws. However, because problems with the procedure surfaced
during the review of 2000 Census operations, the Bureau decided to
combine the emergency and transitional shelter data with the “other non-
institutional group quarters” category that also includes data on people
enumerated in the other categories of the Service-Based Enumeration and
in several other group locations, such as facilities for victims of natural
disasters. As a result of this decision, the Bureau did not separately report
any data from the Service-Based Enumeration in its initial release of
Census 2000 data. These were the only data with separate reporting
categories that the Bureau decided to collapse into another category.

In the fall of 2001, the Bureau produced a special report on the emergency
and transitional shelter data—including most of the same data that the
Bureau earlier stated it could not release because of quality concerns. This
report did not include data on targeted nonsheltered outdoor locations, or
on soup kitchens and mobile food vans. The Bureau added a lengthy
discussion of the limitations of the data and emphasized that they should
not be interpreted as a count of the homeless population.

Although the Bureau worked closely with a number of government entities,
advocacy groups, and other organizations to conduct Service-Based
Enumeration, reconciling its often competing data needs proved
challenging. Compounding the Bureau’s difficulties, expectation gaps
developed between these entities and the Bureau because the Bureau did
not always clearly and consistently communicate its plans.

A key cause of the Bureau’s shifting position on reporting the Service-
Based Enumeration data appears to be its lack of clear, documented, and
consistently applied guidelines governing the release of data from the 2000
Census. Had these guidelines been in place at the time of the census, they



Page 3                                              GAO-03-227 Decennial Census
             could have helped Bureau managers decide whether to release the Service-
             Based Enumeration data and how to characterize these data. Additionally,
             the Bureau could use the guidelines to defend its decisions once they were
             made, thus helping to ensure that the Bureau’s decisions both are, and
             appear to be, completely objective.

             To ensure that the 2010 Census will provide data users with more complete,
             accurate, and useful information on people without conventional housing,
             we recommend that the Secretary of Commerce direct the Bureau of the
             Census to ensure that the procedures for enumerating and estimating
             segments of the population without conventional housing are properly
             tested and evaluated under conditions as similar to the census as possible.
             In addition, the Bureau should develop clearly documented, transparent,
             and consistently applied agencywide guidelines for releasing all census
             data to the public and ensure that plans for releasing data are clearly and
             consistently communicated to the public.

             The Secretary of Commerce forwarded written comments from the Bureau
             of the Census on a draft of this report (see app. I). The Bureau agreed with
             our recommendations and is taking steps to implement them, but took
             exception to our findings concerning the adequacy of its data quality
             guidelines and communication with the public.



Background   The procedures the Bureau used during the 1990 Census to count people
             without conventional housing had limitations that resulted in incomplete
             data.2 To address these limitations and help improve the quality of the data,
             the Bureau used a procedure for the 2000 Census called Service-Based
             Enumeration that attempted to count people where they receive services
             such as emergency shelters, soup kitchens, and regularly scheduled mobile
             food vans. Service-Based Enumeration also counted people in targeted
             nonsheltered outdoor locations such as encampments beneath bridges.
             The operation occurred from March 27 through March 29, 2000.3



             2
               For further information see, U.S. General Accounting Office, 1990 Census: Limitations in
             Methods and Procedures to Include the Homeless, GAO/GGD-92-1 (Washington, D.C.:
             Dec. 30, 1991).
             3
               For information on the conduct of Service-Based Enumeration see, U.S. General
             Accounting Office, 2000 Census: Progress Report on the Mail Response Rate and Key
             Operations, GAO/T-GGD/AIMD-00-136 (Washington, D.C.: Apr. 5, 2000).




             Page 4                                                      GAO-03-227 Decennial Census
              According to Bureau officials, Service-Based Enumeration was not
              designed, and was never intended, to provide a specific count of homeless
              persons. Instead, the operation was part of a larger effort to count people
              without conventional housing, including people in “institutional group
              quarters” such as correctional facilities, nursing homes, and mental
              hospitals, and “non-institutional group quarters” such as college
              dormitories, military quarters, and group homes. Service-Based
              Enumeration counted people in specific categories of noninstitutional
              group quarters.

              To help ensure a complete count of people without conventional housing,
              the Bureau partnered with local governments and community advocacy
              groups to obtain lists of service locations and to assist with the
              enumeration.4 In some cases, the Bureau hired clients of the advocacy
              groups and other people trusted by the homeless to conduct Service-Based
              Enumeration. For example in Atlanta, an advocacy group for homeless
              veterans helped the Bureau employ homeless veterans to improve the
              count of this population. Local governments helped the Bureau as well,
              often investing considerable resources. For example, Los Angeles paid to
              keep its city-run shelters open on the night they were enumerated so that
              people using their services could be counted.



Scope and     To address your concerns about the Bureau’s dissemination of data on
              persons without conventional housing, we agreed to examine (1) the
Methodology   Bureau’s plans for reporting the results of Service-Based Enumeration and
              its reasons for changing those plans and (2) the Bureau’s protocols for
              releasing data. To accomplish these objectives, we interviewed key Bureau
              officials and reviewed relevant Bureau documents and data such as
              operational plans, decision memorandums, and the Bureau’s partnership
              program evaluation.

              In order to obtain the perspective of data users, partners, and stakeholders,
              we conducted in-person and telephone interviews with homeless
              advocates, local government officials, and representatives of public service
              agencies in New York City, Los Angeles, Cleveland, Atlanta, and
              Washington, D.C. These cities had large numbers of people without


              4
                For additional information see, U.S. General Accounting Office, 2000 Census: Review of
              Partnership Program Highlights Best Practices for Future Operations, GAO-01-579
              (Washington, D.C.: Aug. 20, 2001).




              Page 5                                                      GAO-03-227 Decennial Census
                       conventional housing and they were actively involved with the Bureau
                       during the 2000 Census. The organizations we contacted also provided
                       relevant documentation, such as comprehensive file documents relating to
                       partnership activities.

                       In addition to the above locations, we did our audit work at Bureau
                       headquarters in Suitland, Maryland. Our audit work was conducted from
                       April 2002 through September 2002 in accordance with generally accepted
                       government auditing standards.

                       We requested comments on a draft of this report from the Secretary of
                       Commerce. On November 21, 2002, the Secretary forwarded the Bureau’s
                       written comments on the draft (see app. I). We address these comments at
                       the end of this report.



The Bureau of the      Under the Bureau’s original plan for releasing Service-Based Enumeration
                       data in Summary File-1 (SF-1), 5 the emergency and transitional shelter
Census Twice Changed   count was one of several categories of noninstitutional group quarters data
Plans for Reporting    that were to be reported separately. Other people counted in the Service-
                       Based Enumeration, including people counted at targeted nonsheltered
Service-Based          outdoor locations, soup kitchens, and regularly scheduled mobile food
Enumeration Data       vans, were to be combined and reported under the category “other non-
                       institutional group quarters.” This category also included residential care
                       facilities providing protective oversight, shelters against domestic violence,
                       staff dormitories for nurses and interns at military and general hospitals,
                       and living quarters for victims of natural disasters.




                       5
                         The SF-1 is the summary file in which the Bureau presents population and housing data for
                       the total population. Other than the Census 2000 Redistricting Data Summary File, the SF-1
                       file is the first product released after the census.




                       Page 6                                                       GAO-03-227 Decennial Census
This decision was documented in an April 1999 internal memorandum from
the Bureau’s Assistant Division Chief for Special Population Statistics to
the Assistant Division Chief for Census Programs. The Service-Based
Enumeration operation took place a year later, in March 2000. The April
1999 plan was in large part a reaction to the challenges the Bureau faced
counting the emergency shelter and street population during the 1990
Census. Although the Bureau disseminated separate counts of people
found at emergency shelters, preidentified street locations, and similar
sites, the counts proved to be incomplete. 6 Moreover, the Bureau stated in
its October 2001 report that despite its warnings to the contrary, the data
were sometimes misinterpreted as a “homeless” count.

The October report does not offer an example of this, but the
misinterpretation clearly played a role in a lawsuit against the Bureau. 7 As
a result, when designing the 2000 census, the Bureau attempted to both
improve the count and take precautions to ensure that the Service-Based
Enumeration count would not be misconstrued as a count of the homeless.

The Bureau’s data dissemination plans took into account the
recommendations of the Commerce Secretary’s 2000 Census Advisory
Committee, a panel that included representatives of advocacy and other
groups (including representatives from organizations that represent local
governments) that met periodically to review the Bureau’s plans. The
homeless population was represented by the National Coalition for the
Homeless—an advocacy group that coordinates a network of 300 state and
local housing and homeless organizations. In its January 1999 final report,
the Census 2000 Advisory Committee recommended that special attention
be paid to tabulating the results of Service-Based Enumeration and
targeted outdoor enumerations so that they could not be aggregated and
used as a homeless count.



6
  For more information on the 1990 count see, U.S. General Accounting Office, Counting the
Homeless: Limitations of 1990 Census Results and Methodology, GAO/T-GGD-91-29
(Washington, D.C.: May 9, 1991) and 1990 Census: Limitations in Methods and Procedures
to Include the Homeless, GAO/GGD-92-1 (Washington, D.C.: Dec. 30, 1991).
7
  One example was the National Law Center On Homelessness and Poverty, et. al., v. Brown,
where plaintiffs alleged that the Department of Commerce effectively excluded the nation’s
homeless population from the 1990 decennial census in violation of the Constitution and the
Administrative Procedure Act. The District Court denied the plaintiff’s claim (1994 WL
521334) (D.D.C. Sept. 15, 1994) and the Court of Appeals dismissed the lawsuit for lack of
standing. 91 F.3d 178 (D.C. Cir. 1996).




Page 7                                                       GAO-03-227 Decennial Census
The Bureau Changed Its                                         In January 2001, 5 months before the SF-1 release, the Bureau reversed its
Dissemination Plans                                            April 1999 decision to release emergency and transitional shelter data
                                                               separately because of “data quality concerns.” Instead, as shown in figure
Because of Data Quality                                        1, the Bureau planned to combine the emergency and transitional
Concerns


Figure 1: The Bureau Changed Its Original Plan to Release Emergency and Transitional Shelter Data and Combined Them with
Other Noninstitutional Group Quarters Data

                                   Original plan (April 1999)                                                                     Revised plan (January 2001)


                                           Group quarters
                                                                                                              Group quarters




          Institutionalized population                      Non-Institutionalized
                                                              NoninstitutionalizedPopulation
                                                                                     population
              reporting categories                                    reporting categories
          ■ Correctional institutions                       ■ College dorms                                                     Noninstitutionalized population
          ■ Nursing homes                                                                                                            reporting categories
                                                            ■ Military quarters
          ■ Hospitals                                       ■ Group homes                                                    ■ College dorms
          ■ Juvenile institutions                           ■ Religious group quarters                                       ■ Military quarters
                                                            ■ Worker dormitories                                             ■ Group homes
                                                            ■ Crews of maritime vessels                                      ■ Religious group quarters
                                                                                                                             ■ Worker dormitories
                                                            ■ Emergency and transitional shelters
                                                                                                                             ■ Crews of maritime vessels
                                                            ■ Other noninstitutional group quarters
                                                                                                                             ■ Other noninstitutional group quarters




                                 The other noninstitutional group quarters categories are:
                                 Soup kitchens and regularly scheduled food vans                      The other noninstitutional group quarters categories are:
                                 Targeted nonsheltered outdoor locations                              Emergency and transitional shelters
                                 Residential facilities "providing protective oversight"
                                 Living quarters for victims of natural disasters                     Soup kitchens and regularly scheduled food vans
                                 Staff residents of institutions                                      Targeted nonsheltered outdoor locations
                                 Shelters for victims of domestic violence                            Residential facilities "providing protective oversight"
                                                                                                      Living quarters for victims of natural disasters
                                                                                                      Staff residents of institutions
                                                                                                      Shelters for victims of domestic violence
Source: GAO depiction based on Bureau of the Census data.




                                                               Page 8                                                                              GAO-03-227 Decennial Census
shelter data with the “other non-institutional group quarters.” This
category contained data on a variety of living arrangements including
facilities for natural disaster victims. The Bureau’s decision was contained
in an internal Bureau memorandum from the Chief of the Population
Division to the Chief of the Decennial Systems and Contracts Management
Office. Bureau officials told us that the decision to exclude a separate
emergency and transitional shelter count in SF-1 was made between
December 2000 and January 2001, by the Director of the Decennial Census
with input from the Associate Director Decennial Census, the Population
Division, the Associate Director for Demographic Programs, the Decennial
Management Division, and the Decennial Statistical Studies Division.8

According to Bureau officials, their concerns focused on the accuracy of a
new statistical procedure called “multiplicity estimation” that adjusted the
number counted to better reflect the number of actual shelter users.
Because Service-Based Enumeration only counted people who were at
these facilities on the day of enumeration, the Bureau intended to use
multiplicity estimation to calculate the number of people who used these
facilities but were not present during Service-Based Enumeration. The
multiplicity estimation procedure was based on information from those
who were counted and on the number of times they used the service
facilities in the prior week. An estimate of people not counted on the day
of enumeration was added to the count of people who were. According to
the Bureau, the multiplicity estimates tested well during the 1998 dress
rehearsal for the 2000 Census possibly because the three rehearsal sites did
not offer large enough sample sizes of the appropriate populations to
adequately test this procedure.9 However, during the 2000 Census the
Bureau found that a census question pertaining to facility usage upon
which the multiplicity estimates were based had a low response rate.
Moreover, the Bureau found that respondents, particularly in shelters, did
not answer the question accurately. Due to data quality concerns, the
Bureau decided not to use multiplicity estimation to adjust the data and
consequently decided not to report the data separately.


8
 Although the Bureau changed its plan in January 2001, the technical documentation for
SF-1, released at the same time, still indicated that emergency and transitional shelter data
would be separately reported.
9
  The dress rehearsal for the 2000 Census was conducted in Sacramento, California, City of
Columbia, South Carolina, and Menominee County, Wisconsin, including the Menominee
Indian Reservation. The dress rehearsal was designed to test the overall design of the 2000
Census.




Page 9                                                         GAO-03-227 Decennial Census
                            Bureau officials said they did not announce the change in plans because
                            they were still evaluating the problems with the data. It was not until June
                            2000 that the Bureau began recalculating the data and making a final
                            decision on which categories to aggregate. Ultimately, the Bureau did not
                            report any of the Service-Based Enumeration data separately in SF-1.
                            Emergency and transitional shelter data were the only data that were to be
                            released in SF-1 under separate reporting categories that the Bureau
                            decided to combine with another category.



The Bureau Produced a       The release of the SF-1 data in June 2001 produced public discussion in the
Special Report in October   press, among census partners, and in the Congress about the Bureau’s
                            decision to not separately release Service-Based Enumeration data. In a
2001 on the Emergency and
                            briefing for staff of the House Committee on Governmental Affairs, the
Transitional Shelter        Associate Director of the Decennial Census announced that the Bureau
Population                  planned to produce a separate report on the emergency and transitional
                            shelter data. In October 2001, the Bureau issued a special report, entitled
                            Emergency and Transitional Shelter Population: 2000. This report
                            separately identified emergency and transitional shelter data for various
                            levels of geography down to the census tract level with 100 or more people
                            in emergency and transitional shelters. The report did not include data for
                            the populations in targeted nonsheltered outdoor locations, soup kitchens,
                            regularly scheduled mobile food vans, and shelters for domestic violence.
                            The 17-page report contains an extensive discussion on the limitations of
                            the data. For example, the Bureau noted that the data in the report should
                            not be construed as a count of people without conventional housing.
                            Moreover, the emergency and transitional shelter data at the census tract
                            level are not in the hard copy, but rather in the Internet version of the
                            report.10 The Bureau stated that all Census 2000 data at the tract level are
                            available on the Internet and are not available in printed reports.

                            The October report contains most of the same data that were to be released
                            under the April 1999 dissemination plan for SF-1. The Bureau asserted that
                            the data quality concerns with the emergency and transitional shelter data
                            (cited when it changed the plan to release these data in SF-1) required that
                            the data be presented in a manner that allowed the Bureau to clearly



                            10
                              U.S. Bureau of the Census, Population in Emergency and Transitional Shelters
                            (PHC-T-12) (Washington, D.C.: Oct. 30, 2001).
                            http://www.census.gov/population/www/cen2000/phc-t12.html




                            Page 10                                                   GAO-03-227 Decennial Census
                            outline the data’s limitations. The October 2001 report contained an
                            extended discussion of these limitations.

                            The October 2001 report also identified reasons the Bureau did not (and
                            never planned to) separately release data on people counted at targeted
                            nonsheltered outdoor locations, soup kitchens, regularly scheduled mobile
                            food vans, and shelters for victims of domestic violence, including the
                            following.

                            • People without conventional housing who were at outside locations
                              other than the targeted nonsheltered outdoor locations identified for the
                              census were not included in the TNSOL operation.

                            • For the purposes of the TNSOL operation, the definition of “outdoor”
                              excluded both mobile and transient locations used by people
                              experiencing homelessness as well as abandoned buildings.

                            • The option was given to the individuals found at soup kitchens and
                              regularly scheduled mobile food vans to select “usual home elsewhere.”
                              For example, if an individual enumerated at a soup kitchen listed a usual
                              home elsewhere, then that person was tabulated at their usual residence
                              and not at the service location. Therefore, the data on this population
                              would not reflect a true count of the individuals using these services.

                            Prior to publicly releasing the October report, the Bureau asked two
                            representatives from the National Coalition for the Homeless to review a
                            draft of the portion of the report that described the limitations of the data.
                            The National Coalition for the Homeless commented extensively on the
                            section containing the caveats and limitations in order to strengthen the
                            report. A member of the Board of Directors for the National Coalition for
                            the Homeless told us that he provided this feedback both as an
                            academician and a stakeholder. Bureau officials stated that because of its
                            position on the Bureau’s Census Advisory Committee, the National
                            Coalition for the Homeless was the only advocacy group that reviewed any
                            portion of the October 2001 report prior to its publication.



Meeting Data Users’ Needs   The controversy surrounding the release of the combined Service-Based
Proved Challenging          Enumeration data highlights the challenges the Bureau faced in 2000 trying
                            to meet the needs of various data users and the work the Bureau still needs
                            to do when planning for the 2010 Census to better reconcile those needs.
                            For example, several organizations we contacted favored the separate



                            Page 11                                             GAO-03-227 Decennial Census
release of the Service-Based Enumeration data categories. Indeed, local
government officials we talked to in New York City believed that the data
would help with grant applications, projections about future service needs,
and determining their success in getting people off the streets and into
shelters. The Executive Director of the Northeast Ohio Coalition for the
Homeless stated that the city of Cleveland does not do its own count of this
population and, therefore, the Bureau numbers are the only ones available
on this segment of the population. Los Angeles city officials wanted the
Service-Based Enumeration data so they could better target their services
and, like Cleveland, Los Angeles did not have its own data. Several of these
entities stated that the potential misuse of data was not a valid reason for
not separately releasing data.

In addition, the majority of the organizations we contacted partnered with
the Bureau anticipating that they would be able to use the Service-Based
Enumeration data to evaluate whether improvements were made in
enumerating local populations without conventional housing in 2000
compared to 1990. The Assistant City Attorney of Los Angeles estimated
that Los Angeles spent about $300,000 on the effort to improve the count of
Los Angeles’s people without conventional housing. For example, as part
of an extensive effort to help the Bureau develop a list of targeted
nonsheltered outdoor locations, the city provided senior Bureau staff with
a helicopter tour over some outdoor locations where people without
conventional housing lived. The Assistant City Attorney of Los Angeles
stated that she believed the city would get the targeted nonsheltered
outdoor locations data that they helped collect and wanted to review. In
addition, because of the Bureau’s focus on counting people at shelters, the
city kept shelters open on the night of the enumeration at its own expense
even though shelters in Los Angeles typically do not have many people
during warm weather. Los Angeles expected to have detailed data to use to
evaluate the effectiveness of its resource allocation.

However, the National Coalition for the Homeless and other advocates of
the homeless opposed the separate release of any of the Service-Based
Enumeration data. They were concerned that these data could be misused
as a count of the homeless population and lead to flawed decision-making
by policymakers.

Ultimately, the Bureau left a number of data users unsatisfied. Those who
wanted the Service-Based Enumeration categories released separately did
not feel the Bureau met their expectations with the data released in SF-1 or
with the release of the October report. Users who opposed the separate



Page 12                                            GAO-03-227 Decennial Census
release of the data and were pleased that SF-1 combined the Service-Based
Enumeration components with other data were displeased that the October
2001 report was released.

The difficulties the Bureau experienced trying to reconcile the competing
needs and interests of data users illustrates the importance of effective
communication between the Bureau and its key data users and partners to
ensure no expectation gaps develop. More than just a good business
practice, federal internal control standards require agencies to have
effective external communications with groups that can have a serious
impact on programs, projects, operations, and other activities.11

However, our conversations with several Bureau partners and our review
of Bureau documents suggest that communications were sometimes vague
and insufficient. For example, although the April 1999 memorandum that
outlined the Bureau’s initial data dissemination plans was written a year
before the 2000 Census, this information may not have been effectively
communicated to the Bureau’s partners. Indeed, at a Capitol Hill briefing
on this topic in June 2001, Bureau officials themselves acknowledged that
they did not do a good job of communicating on this issue. Some of the
partners we spoke to indicated that had they known earlier about the
Bureau’s plans to limit the release of Service-Based Enumeration data they
might have focused their resources on different census operations.
Further, our review of Bureau documents indicated that the information on
the “official plan” for the release of the different Service-Based
Enumeration categories of data was limited and inconsistent. Some
partners stated that they did not know that the Bureau never intended to
report the targeted nonsheltered outdoor location data.

Although the Bureau made numerous presentations on Service-Based
Enumeration that emphasized there would be no count of the homeless,
the Bureau provided little detail on how components of Service-Based
Enumeration would actually be presented. In the absence of clear
communication from the Bureau, partners developed their own
expectations of what would be released. Several of the local officials and
advocates that we spoke to expected that the data would be released in the


11
  U.S. General Accounting Office, Internal Control Standards, Internal Control
Management and Evaluation Tool (August 2001) and U.S. General Accounting Office,
Internal Control Standards for Internal Control in the Federal Government (November
1999).




Page 13                                                   GAO-03-227 Decennial Census
                        same detail as it was in 1990, because they were not told otherwise. For
                        example, a Los Angeles government official said that the Bureau stated it
                        would not provide a homeless count in 1990, but it still released the street
                        count data separately. By focusing resources on counting specific
                        categories of the population, the Bureau may have created expectations
                        that there would be a count of that population.



Census Bureau Had       A cause of the Bureau’s shifting position on reporting the components of
                        Service-Based Enumeration appears to be its lack of documented, clear,
Few Documented          transparent, and consistently applied guidelines governing the release of
Guidelines Governing    data from the 2000 Census. Except for some guidance aimed at protecting
                        the confidentiality of census records, the Bureau had few written
the Release of Census   guidelines on the level of quality needed to release data to the public. Had
Data                    these guidelines been in place during the decennial census, they could have
                        informed the Bureau’s decision on whether to release the Service-Based
                        Enumeration data, how to characterize these data, and help defend the
                        decision after it was made. Such guidelines could also provide a basis
                        ahead of time for expectations about the conditions under which data will
                        or will not be released.

                        Although Bureau officials emphasized that the Bureau has a long tradition
                        of high standards and procedures that yield quality data (to its credit, the
                        Bureau’s quality assurance practices identified the problem with the
                        multiplicity estimator), the officials acknowledged that these standards
                        were primarily part of the agency’s institutional knowledge. The written
                        guidance that did exist appeared to be vague and insufficient for making
                        consistent decisions on the quality thresholds needed for releasing data to
                        the public, and the circumstances under which it might be appropriate to
                        suppress certain data.




                        Page 14                                             GAO-03-227 Decennial Census
                           According to the Bureau’s Associate Director for Methodology and
                           Standards, a technical paper issued in 1974 and revised in 1987 contained
                           the Bureau’s only written guidelines for discussion and presentation of
                           errors in data. This paper noted that, “[e]stimates for individual cells of a
                           published table should not be suppressed solely because they are subject to
                           large sampling errors or large nonsampling variances, provided users are
                           given adequate caution of the lack of reliability of the data. On the other
                           hand, data known to have very serious bias may be suppressed.”12



A Newly Created Bureau     In 2000, the Bureau initiated a new quality assurance program to document
Program Could Provide      Bureau-wide protocols designed to ensure the quality of data collected and
                           disseminated by the Bureau. The Bureau’s Methodology and Standards
Guidelines for Releasing
                           Council is charged with setting statistical and survey quality standards and
Data                       guidelines for Bureau surveys and censuses. In support of this role, the
                           council has established a quality framework in which the demographic,
                           economic, and decennial areas can share and support common principles,
                           standards, and guidelines. The quality framework covers eight unique
                           areas, one of which is dissemination.

                           Because this Bureau program is in its initial stages, we could not evaluate
                           it. However, Bureau officials believe that the program is a significant first
                           step in addressing the lack of agencywide written guidelines for releasing
                           data. The initiative appears to be consistent with Office of Management
                           and Budget guidelines issued in February 2002 requiring federal agencies to
                           issue their own guidance for ensuring and maximizing the quality,
                           objectivity, utility, and integrity of information disseminated by the
                           agency.13 As the Bureau develops its guidelines, it will be important that
                           they be well documented, transparent, clearly defined, and consistently
                           applied.



Conclusions                Although Service-Based Enumeration was designed to address the
                           challenges the Bureau encountered during the 1990 Census in obtaining a
                           complete count of people without conventional housing, the Bureau’s
                           experience during the 2000 Census suggests that tallying this population

                           12
                             U.S. Department of Commerce, Technical Paper 32, Standards for Discussion and
                           Presentation of Errors in Data (March 1974), p. 3.
                           13
                                Issued at 67 Fed. Reg. 8452 (Feb. 22, 2002).




                           Page 15                                                  GAO-03-227 Decennial Census
                      group remains problematic. Moreover, the Bureau’s difficulties were
                      compounded by its shifting position on how to report the data once they
                      were collected. A number of government, community, and advocacy
                      organizations helped the Bureau enumerate this population group.
                      However, the Bureau, by first planning to release the data one way, then
                      changing the decision, and ultimately releasing the data anyway—all for
                      reasons that were not clearly articulated to the Bureau’s stakeholders—
                      raised questions about the Bureau’s decision-making on data quality issues.
                      As noted at the beginning of this report, related questions have also been
                      raised about how the Bureau collected and reported data on Hispanic
                      subgroups. To the extent similar incidents occur in the future, they could
                      undermine public confidence in the accuracy and credibility of Bureau
                      data.

                      Thus, as the Bureau plans for the 2010 Census, it will be important for it to
                      refine its methods for enumerating people living in unconventional housing
                      and reporting the resulting data, in part by properly testing and evaluating
                      those methods. As noted earlier, the Bureau could not properly test a key
                      statistical technique during the census dress rehearsal because the sample
                      size was too small.

                      Moreover, while addressing the competing needs and desires of data users
                      will likely remain a considerable challenge, it will be important for the
                      Bureau to more effectively articulate its plans to avoid the expectation gaps
                      that occurred during 2000. The Bureau’s plans for collecting data on
                      persons without conventional housing need to specify how the Bureau
                      plans to separately report these data.

                      Bureau-wide guidelines on the level of quality needed to release data to the
                      public, on how and when to document data limitations, and on the
                      circumstances under which it is acceptable to suppress data, could help the
                      Bureau be more accountable and consistent in its dealings with data users
                      and stakeholders, and help ensure that the Bureau’s decisions both are, and
                      appear to be, totally objective.



Recommendations for   To ensure that the 2010 Census will provide public data users with more
                      complete, accurate, and useful information on the segment of the
Executive Action      population without conventional housing, we recommend that the
                      Secretary of Commerce direct the Director of the Bureau of the Census to
                      do the following.




                      Page 16                                             GAO-03-227 Decennial Census
                      1. Ensure that all procedures for enumerating and estimating segments of
                         the population without conventional housing are properly tested and
                         evaluated under conditions as similar to the census as possible.

                      2. Develop agencywide guidelines for Bureau decisions on the level of
                         quality needed to release data to the public, how to characterize any
                         limitations in the data, and when it is acceptable to suppress the data
                         for reasons other than protecting the confidentiality of respondents.
                         Ensure that these guidelines are documented, transparent, clearly
                         defined, and consistently applied.

                      3. Ensure that the Bureau’s plans for releasing data are clearly and
                         consistently communicated with the public.



Agency Comments and   The Secretary of Commerce forwarded written comments from the Bureau
                      of the Census on a draft of this report (see app. I). The Bureau agreed with
Our Evaluation        each of our recommendations and, as indicated in the letter, is taking steps
                      to implement them. However, it expressed several general concerns about
                      our findings. The Bureau’s principal concerns and our response are
                      presented below. The Bureau also suggested minor wording changes to
                      provide additional context and clarification. We accepted the Bureau’s
                      suggestions and made changes to the text as appropriate.

                      The Bureau took exception to our findings concerning the adequacy of its
                      data quality guidelines, noting that the Bureau’s decisions regarding the
                      release and characterization of emergency and transitional shelter data
                      were based on established guidelines for data quality. However, the Bureau
                      did not cite any written guidelines to support its position. As noted in our
                      report, Bureau officials, including the Associate Director for Methodology
                      and Standards, told us that the Bureau had few written guidelines,
                      standards, or procedures related to the quality of data released to the
                      public. In this report we acknowledge the Bureau’s tradition of high
                      standards and procedures that yield quality data. However, according to
                      the Bureau, these standards are generally undocumented and part of the
                      agency’s institutional knowledge. To provide a basis for consistent
                      decision-making and clear communication within the Bureau and to the
                      public, guidelines on the quality of data released to the public must be fully
                      documented, transparent, clearly defined, and consistently applied.

                      Additionally, the Bureau said that when data do not meet an acceptable
                      level of quality, it considers various options for modifying its dissemination



                      Page 17                                             GAO-03-227 Decennial Census
plans. The Bureau’s decision to delay the release of the emergency and
transitional shelter data may have been entirely appropriate. Our concern
is not that the Bureau changed its plans, but that it could not provide us its
guidelines for determining an acceptable level of quality or clearly indicate
how it determined that the data did not meet minimal quality standards for
release in SF-1.

The Bureau also commented that its decisions regarding the distribution of
data from SF-1 were well publicized and that the only change in Bureau
plans for the release of Service-Based Enumeration data was the decision
to delay release of the emergency and transitional shelter data. This report
focused on the changing plans for the release of the emergency and
transitional shelter data and noted that the Bureau never intended to
release any other data from the Service-Based Enumeration. However, we
found that the Bureau did not effectively communicate its decisions with
its partners or the public. Decisions on the release of the emergency and
transitional shelter data were contained in internal decision memoranda.
We found that these decisions were not always reflected in new releases of
the SF-1 documentation. Although Bureau officials told us that they always
intended to produce a separate report on emergency and transitional
shelter data, they did not make this intention public when the SF-1 data
were released. Some stakeholders did not realize that the Bureau was not
releasing emergency and transitional shelter data with SF-1 until they
examined the SF-1 data. As we stated in our report, these communication
problems can undermine stakeholder and public confidence in the Bureau
and its products.

As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from its
issue date. At that time, we will send copies of this report to the Chairman
of the House Committee on Government Reform, the Chairman of the
Subcommittee on Civil Service, Census and Agency Organization, the
Secretary of Commerce, and the Director of the Bureau of the Census.
Copies will be made available to others on request. This report will also be
available at no charge on GAO’s home page at http://www.gao.gov.




Page 18                                             GAO-03-227 Decennial Census
Please contact me on (202) 512-6806 or by e-mail at daltonp@gao.gov if you
have any questions. Other key contributors to this report were Robert
Goldenkoff, Timothy Wexler, Elizabeth Powell, Chris Miller, James
Whitcomb, Ty Mitchell, Robert Parker, and Michael Volpe.




Patricia A. Dalton
Director
Strategic Issues




Page 19                                           GAO-03-227 Decennial Census
Appendix I

Comments from the Secretary of Commerce                   AA
                                                           ppp
                                                             ep
                                                              ned
                                                                n
                                                                x
                                                                id
                                                                 e
                                                                 x
                                                                 Iis




             Page 20            GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 21                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 22                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 23                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 24                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 25                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 26                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 27                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 28                                   GAO-03-227 Decennial Census
Appendix I
Comments from the Secretary of Commerce




Page 29                                   GAO-03-227 Decennial Census
Related GAO Products on the Results of the
2000 Census and Lessons Learned for a More
Effective Census in 2010
              2000 Census: Refinements to Full Count Review Program Could Improve
              Future Data Quality. GAO-02-562. Washington, D.C.: July 3, 2002.

              2000 Census: Coverage Evaluation Matching Implemented As Planned,
              but Census Bureau Should Evaluate Lessons Learned. GAO-02-297.
              Washington, D.C.: March 14, 2002.

              2000 Census: Best Practices and Lessons Learned for a More Cost-
              Effective Nonresponse Follow-Up. GAO-02-196. Washington, D.C.:
              February 11, 2002.

              2000 Census: Coverage Evaluation Interviewing Overcame Challenges,
              but Further Research Needed. GAO-02-26. Washington, D.C.: December 31,
              2001.

              2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control
              Weaknesses at the U.S. Census Bureau. GAO-02-30. Washington, D.C.:
              December 28, 2001.

              2000 Census: Significant Increase in Cost Per Housing Unit Compared
              to 1990 Census. GAO-02-31. Washington, D.C.: December 11, 2001.

              2000 Census: Better Productivity Data Needed for Future Planning and
              Budgeting. GAO-02-4. Washington, D.C.: October 4, 2001.

              2000 Census: Review of Partnership Program Highlights Best Practices
              for Future Operations. GAO-01-579. Washington, D.C.: August 20, 2001.

              Decennial Censuses: Historical Data on Enumerator Productivity Are
              Limited. GAO-01-208R. Washington, D.C.: January 5, 2001.

              2000 Census: Information on Short- and Long-Form Response Rates.
              GAO/GGD-00-127R. Washington, D.C.: June 7, 2000.




(450102)      Page 30                                        GAO-03-227 Decennial Census
GAO’s Mission            The General Accounting Office, the investigative arm of Congress, exists to
                         support Congress in meeting its constitutional responsibilities and to help improve
                         the performance and accountability of the federal government for the American
                         people. GAO examines the use of public funds; evaluates federal programs and
                         policies; and provides analyses, recommendations, and other assistance to help
                         Congress make informed oversight, policy, and funding decisions. GAO’s
                         commitment to good government is reflected in its core values of accountability,
                         integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO
                         Mailing Lists” under “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300
Address Service Requested