oversight

2000 Census: Coverage Measurement Programs' Results, Costs, and Lessons Learned

Published by the Government Accountability Office on 2003-01-29.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

               United States General Accounting Office

GAO            Report to the Chairman, Subcommittee
               on Civil Service, Census, and Agency
               Organization, Committee on Government
               Reform, House of Representatives

January 2003
               2000 CENSUS
               Coverage
               Measurement
               Programs’ Results,
               Costs, and Lessons
               Learned




GAO-03-287
               a
                                               January 2003


                                               2000 CENSUS

                                               Coverage Measurement Programs’
Highlights of GAO-03-287, a report to          Results, Costs, and Lessons Learned
Chairman, Subcommittee on Civil Service,
Census, and Agency Organization,
Committee on Government Reform,
House of Representatives




To help measure the quality of the             As shown below, the two programs the Bureau employed to measure the
2000 Census and to possibly adjust             quality of the 2000 Census population data did not meet their objectives.
for any errors, the U.S. Census
Bureau (Bureau) conducted the
                                               Coverage Measurement Programs Did Not Achieve Objectives
Accuracy and Coverage Evaluation                                                Objectives
(A.C.E.) program. However, after                Program/objectives              met?         Reasons
obligating around $207 million for              I.C.M.
A.C.E. and its predecessor                      •    Measure census coverage            No           Program was canceled following January
program, Integrated Coverage                                                                         1999 U.S. Supreme Court ruling that the
Measurement (I.C.M.), from fiscal               •    Generate data for                 No            Census Act prohibits the use of sampling to
years 1996 through 2001, the                         apportionment, redistricting, and               apportion seats in the U.S. House of
Bureau did not use either program                    federal programs using                          Representatives.
                                                     statistical sampling and
to adjust the census numbers.                        estimation
Concerned about the amount of
money the Bureau spent on I.C.M.                •   Produce a “one-number”              No
and A.C.E. programs and what was                    census
produced in return, the                         A.C.E.
subcommittee asked us to review                 •    Measure census coverage            No           Uncertainties surrounding the accuracy of
the objectives and results of the                                                                    the A.C.E. results and the inability to
programs, the costs of consultants,             •    Generate data needed for           No           resolve them in time to meet legally
                                                     redistricting and other purposes                mandated deadlines for releasing data.
and how best to track future                         using statistical methods
coverage measurement activities.               Source: GAO.
                                               Note: This table reflects GAO’s analysis of U.S. Census Bureau data.

                                               The A.C.E. program achieved results other than those laid out in the
The Secretary of Commerce should               Bureau’s formal objectives that highlight important lessons learned. They
direct the Bureau to (1) work with             include (1) developing a coverage measurement methodology that is both
Congress and other stakeholders                operationally and technically feasible, (2) determining the level of geography
and soon decide on whether and                 at which coverage measurement is intended, (3) keeping stakeholders,
how coverage measurement will be               particularly Congress, informed of the Bureau’s plans, and (4) adequately
used in 2010, (2) adopt lessons
                                               testing coverage measurement methodologies. It will be important for the
learned from its 2000 Census
experience, and (3) ensure that its            Bureau to consider these as its current plans for the 2010 Census include
financial management systems can               coverage evaluation to measure the accuracy of the census but not
capture and report program                     necessarily to adjust the results.
activities early and that projects’
costs are monitored. The Bureau                Of the roughly $207 million the Bureau obligated for I.C.M./A.C.E. programs
agreed with our recommendations                from fiscal years 1996 through 2001, we identified about $22.3 million that
but noted that for the 2000 Census,            was obligated for contracts involving over 170 vendors. We could not
it followed the steps we identified            identify any obligations prior to 1996 in part because the Bureau included
as lessons learned. It also took               them with its general research and development efforts and did not assign
exception to how we presented our              the I.C.M./A.C.E. operations unique project codes in its financial
conclusions concerning its ability
                                               management system. To track these costs in the future, it will be important
to properly classify certain costs.
                                               for the Bureau to (1) have a financial management system that has specific
                                               project codes to capture coverage measurement costs, (2) establish the
www.gao.gov/cgi-bin/getrpt?GAO-03-287.         project codes as early in the planning process as possible, and (3) monitor
To view the full report, including the scope   the usage of the codes to ensure that they are properly charged.
and methodology, click on the link above.
For more information, contact Patricia A.
Dalton at (202) 512-6806 or
daltonp@gao.gov.
Contents



Letter                                                                                                 1
                           Results in Brief                                                            2
                           Background                                                                  4
                           Scope and Methodology                                                       5
                           Coverage Measurement
                              Programs for the 2000 Census
                              Did Not Meet Bureau Objectives                                          7
                           I.C.M./A.C.E. Contractor Costs Are Not Complete                           13
                           Tracking Future Coverage Measurement Costs                                15
                           Conclusions                                                               16
                           Recommendations for Executive Action                                      17
                           Agency Comments and Our Evaluation                                        17


Appendix
             Appendix I:   Comments from the Department of Commerce                                  20


Related GAO Products                                                                                 23


Tables                     Table 1: Coverage Measurement Programs Did Not Achieve
                                    Objectives                                                         7
                           Table 2: Census 2000 I.C.M./A.C.E. Contractor Costs for Fiscal
                                    Years 1996 through 2001 (Dollars in thousands)                   14




                           Page i                        GAO-03-287 Census Coverage Measurement Programs
A
United States General Accounting Office
Washington, D.C. 20548



                                    January 29, 2003                                                                          Leter




                                    The Honorable Dave Weldon, M.D.
                                    Chairman
                                    Subcommittee on Civil Service,
                                      Census, and Agency Organization
                                    Committee on Government Reform
                                    House of Representatives

                                    Dear Mr. Chairman:

                                    To assess the quality of population data for the 2000 Census and to possibly
                                    adjust for any errors, the U.S. Census Bureau (Bureau) conducted the
                                    Accuracy and Coverage Evaluation (A.C.E.) program. The A.C.E. program
                                    was first included in Bureau program documents in November 1998 and
                                    funded for fiscal years 2000 through 2002 with proposed funding through
                                    December 31, 2002. Its predecessor, Integrated Coverage Measurement
                                    (I.C.M.), began in May 1995 and was funded by the Bureau for fiscal years
                                    1996 through 1999. The Bureau obligated about $207 million to both
                                    programs from fiscal years 1996 through 2001,1 which was about 3 percent
                                    of the $6.5 billion total estimated cost of the 2000 Census. However, neither
                                    program was used to adjust the 2000 Census population count.

                                    Concerned about the amount of money the Bureau spent on both programs
                                    as well as what was received in return for its investment, you and former
                                    Vice Chairman Dan Miller asked us to examine (1) the Bureau’s objectives
                                    for the I.C.M./A.C.E. programs and the extent to which those objectives
                                    were met, (2) the cost of consultants and technical studies, and (3) ways to
                                    track the costs of coverage measurement activities in future censuses. This
                                    report responds to that request.

                                    In October 2002, we issued a report that provides additional information on
                                    the cost of the I.C.M./A.C.E. programs.2 Both reports are part of our
                                    ongoing series on the results of the 2000 Census and the lessons learned for
                                    planning a more cost-effective census in 2010. (See the Related GAO



                                    1
                                     At the time of our report, obligated costs after fiscal year 2001 were not final.
                                    2
                                    U.S. General Accounting Office, 2000 Census: Complete Costs of Coverage Evaluation
                                    Programs Are Not Available, GAO-03-41 (Washington, D.C.: Oct. 31, 2002).




                                    Page 1                                  GAO-03-287 Census Coverage Measurement Programs
                   Products section at the end of this report for the assessments issued to
                   date.)



Results in Brief   The coverage measurement programs the Census Bureau planned for the
                   2000 Census—I.C.M. and its successor, A.C.E.—did not meet their intended
                   objectives. Although the Bureau designed I.C.M. to measure census
                   coverage; generate data for apportionment, redistricting, and federal
                   programs using statistical sampling and estimation; and produce a “one-
                   number” census based on statistical sampling and estimation, the Bureau
                   abandoned the program following a Supreme Court ruling that the Census
                   Act prohibited the use of statistical sampling to generate population data
                   for reapportioning the House of Representatives.3 Because its
                   replacement—A.C.E.—did not provide a reliable measure of census
                   accuracy in time to meet legally mandated deadlines for releasing
                   redistricting data, the Bureau decided against using it to adjust the census
                   data for nonapportionment purposes.

                   The difficulties the Bureau encountered in trying to implement I.C.M. and
                   A.C.E. underscore the importance of (1) developing a coverage
                   measurement methodology that is both technically and operationally
                   feasible, (2) determining the level of geography at which coverage
                   measurement is intended, (3) keeping stakeholders, particularly Congress,
                   informed of the Bureau’s plans, and (4) adequate testing. It will be
                   important for the Bureau to address these lessons learned as its planning
                   efforts for the 2010 Census continue. Those plans currently call for a
                   coverage measurement program to evaluate the accuracy of the census,
                   whereas the issue of whether coverage measurement will be used to adjust
                   the numbers has not yet been resolved.

                   Concerning the cost of contracts, including the consultants and technical
                   studies related to the I.C.M./A.C.E. programs, we identified about
                   $22.3 million of obligated amounts for over 170 vendors for fiscal year 1996
                   through fiscal year 2001 from unaudited Bureau financial management
                   reports. This amount does not represent the complete contract costs for
                   the I.C.M./A.C.E. programs for three reasons. First, the Bureau considered
                   costs from earlier years to be part of its general research and development
                   efforts and did not assign unique project codes to identify I.C.M./A.C.E.


                   3
                    Department of Commerce v. United States House of Representatives, 525 U.S. 316 (1999).




                   Page 2                               GAO-03-287 Census Coverage Measurement Programs
programs and related costs in its financial management system. Second,
although $182,000 of fiscal year 1996 obligated costs were identified in the
Bureau’s financial management systems as contract costs for an I.C.M.
special test, the Bureau did not consider these costs as part of the
I.C.M./A.C.E. programs and classified these costs as general research. We
disagreed with the Bureau on this point and have included this amount in
our report as part of the I.C.M./A.C.E. contract costs that we could identify
from Bureau records. Finally, certain costs, such as program evaluations,
were Bureau-wide in nature, and the portion attributable to I.C.M./A.C.E
could not be separated out.

The Bureau’s ability to track future costs of coverage measurement
activities is primarily dependent on its ability to (1) ensure that its financial
management system accurately and completely captures the accounting
and reporting of project codes, (2) design its project codes to capture
coverage measurement activities as early in the planning process as
possible, even though the activities’ names may change as the programs
evolve, and (3) correctly charge the project codes established.

Although the Bureau has never used the results of its coverage
measurement programs to adjust census numbers, we believe that an
evaluation of the accuracy of the census is essential given the importance
of the data, the need to know the nature of any errors, and the cost of the
census overall. Whether the results of the evaluation should be used to
adjust the census is still an open question, the answer to which should
involve discussions between the Bureau, Congress, and other stakeholders,
and be based on detailed data and a convincing demonstration of the
feasibility of the Bureau's proposed approach. Regardless of the outcome
of the decision, it is critical that it be made soon so that the Bureau can
proceed with its planning. The longer the Bureau goes without a firm
decision on the role of coverage measurement, the greater the risk of
wasted resources and disappointing results. In light of the challenges
facing the Bureau as it prepares for the next decennial census in 2010, we
recommend that the Secretary of Commerce direct the U.S. Census Bureau
to

• in conjunction with Congress and other stakeholders, decide soon on
  whether and how coverage measurement will be used in the 2010
  Census;

• consider incorporating lessons learned from its coverage measurement
  experience during the 2000 Census, such as (1) demonstrating both the



Page 3                            GAO-03-287 Census Coverage Measurement Programs
                operational and technical feasibility of its coverage measurement
                methods, (2) determining the level of geography at which coverage can
                be reliably measured, (3) keeping Congress and other stakeholders
                informed of its plans, and (4) adequately testing coverage measurement
                prior to full implementation; and

             • ensure that the Bureau’s financial management systems can capture and
               report program activities early in the decennial process and that project
               costs are monitored for accuracy and completeness.

             The Secretary of Commerce forwarded written comments from the Bureau
             of the Census on a draft of this report (see app. I). The Bureau agreed with
             our recommendations noting that they were important steps that should be
             followed in the development of a coverage measurement methodology for
             the 2010 Census. However, the Bureau maintained that most of these steps
             such as keeping Congress and other stakeholders informed of its plans
             were followed for the 2000 Census. We disagree because, as noted in the
             report, the lack of information contributed to stakeholders’ skepticism
             surrounding the Bureau’s plans. The Bureau also took exception to the way
             we presented our conclusions concerning its ability to properly classify
             certain costs associated with the development of the Bureau’s coverage
             measurement programs. Our perspective on the Bureau’s position is
             detailed in the “Agency Comments and Our Evaluation” section at the end
             of this report.



Background   The Bureau puts forth tremendous effort to conduct a complete and
             accurate count of the nation’s population. However, some degree of error in
             the form of persons missed or counted more than once is inevitable
             because of limitations in census-taking methods. Because census results
             are used, among other purposes, to apportion Congress, redraw
             congressional districts, and allocate federal aid to state and local
             governments, the size and demographic composition of these coverage
             errors have become increasingly sensitive since the Bureau was first able
             to generate detailed data on them during the 1980 Census. However, the
             Bureau has never used the results of its coverage measurements to correct
             estimated coverage errors.

             The Bureau first attempted to measure the accuracy of the census in the
             1940s when it compared the census numbers to birth and death certificates
             and other administrative data using a procedure called demographic
             analysis. Modern coverage measurement began with the 1980 Census when



             Page 4                          GAO-03-287 Census Coverage Measurement Programs
              the Bureau compared census figures to the results of an independent
              sample survey of the population. Using statistical methods, the Bureau
              generated detailed measures of the differences among undercounts of
              particular ethnic, racial, and other groups. In the months that followed,
              many lawsuits were filed, most contending that the results of the 1980
              coverage measurement should have been used to adjust the census.
              However, the Bureau designed the evaluation to measure errors, not to
              correct the census results, and the Director of the Census Bureau decided
              against adopting the adjusted numbers, as they were deemed flawed due to
              missing and inaccurate data.

              The quality of the coverage measurement data improved for the 1990
              Census, and the Bureau recommended statistically adjusting the results.
              However, the Secretary of Commerce determined that the evidence to
              support an adjustment was inconclusive and decided not to adjust the 1990
              Census. The adjustment decision was complicated by the fact that the 1990
              Census figures had already been released when the coverage measurement
              results became available in the spring of 1991. The Secretary of Commerce
              was concerned that two sets of numbers—the actual census results and the
              adjusted figures—could create confusion and might allow political
              considerations to play a part in choosing between sets of numbers when
              the outcome of the choices, such as congressional apportionment, could be
              known in advance of a decision.



Scope and     To determine the objectives of 2000 Census I.C.M./A.C.E. programs and
              their results, we reviewed Bureau and other documents that included
Methodology   Federal Register notices; Census Operational Plans; reports to Congress;
              internal memorandums; research and feasibility studies; and reports of the
              Executive Steering Committee for Accuracy and Coverage Policy (ESCAP)
              I and II, which assessed the results of the A.C.E. program and
              recommended how they should be used.

              To determine costs for consultants and technical studies for 2000 Census
              I.C.M./A.C.E. programs, we focused on object class code 25 from the
              financial management reports to obtain contract data. With Bureau
              assistance, we identified I.C.M./A.C.E. project accounts and analyzed
              amounts by fiscal year using the financial management reports generated
              by the Department of Commerce’s Administrative Management System
              (CAMS). We reviewed and analyzed obligated and expended data for all
              coverage measurement programs that existed during the 2000 Census for




              Page 5                         GAO-03-287 Census Coverage Measurement Programs
fiscal years 1991 to 2003. We did not audit financial data provided by the
Bureau.

To determine ways to track future costs, we reviewed current Bureau
financial management reports and considered established standards of
accounting, auditing, and internal controls.

In addition, we met with key Bureau officials to discuss the results of our
analysis and obtain their observations and perspectives. The limitations we
encountered in the scope of our work on this assignment are as follows.

• We were unable to determine the complete contractual and technical
  studies costs of the I.C.M./A.C.E. programs because the Bureau
  considered any I.C.M./A.C.E.-related costs from fiscal years 1991
  through 1995 as part of its general research and development programs
  and thus did not separately track these costs. Although some costs were
  tracked in fiscal year 1996, the Bureau still considered these costs as
  research and development and did not include these costs as
  I.C.M./A.C.E. program costs.

• We were unable to identify I.C.M./A.C.E. portions of costs from projects
  that covered the entire census, such as the 2000 Census Evaluation
  program.

• We did not evaluate the propriety of contracts for I.C.M./A.C.E.
  programs.

Our work was performed in Washington, D.C., and at U.S. Census Bureau
headquarters in Suitland, Maryland, from June 2002 through October 2002
in accordance with generally accepted government auditing standards. On
January 7, 2003, the Secretary of Commerce provided written comments on
a draft of this report. We address these comments in the “Agency
Comments and Our Evaluation” section, and have reprinted them in
appendix I.




Page 6                          GAO-03-287 Census Coverage Measurement Programs
Coverage                  In planning the 2000 Census, the Bureau developed a new coverage
                          measurement program, I.C.M., that was designed to address the major
Measurement               shortcomings of the 1990 coverage measurement program. However, as
Programs for the 2000     shown in table 1, much like similar programs in earlier censuses, the
                          Bureau did not use I.C.M. and its successor program, A.C.E., to adjust the
Census                    census because of legal challenges, technical obstacles, and the inability to
Did Not Meet Bureau       resolve uncertainties in the data in time to meet the deadlines for releasing
Objectives                the data.



                          Table 1: Coverage Measurement Programs Did Not Achieve Objectives

                                                                  Objectives
                          Program/objectives                      met?           Reasons
                          I.C.M.

                          • Measure census coverage.              No.            Program was canceled following
                                                                                 January 1999 U.S. Supreme Court
                          • Generate data for apportionment, No.                 ruling that the Census Act prohibits the
                            redistricting, and federal program                   use of sampling to produce counts
                            purposes using statistical                           used to apportion seats in the U.S.
                            sampling and estimation.                             House of Representatives. The Bureau
                                                                                 then replaced I.C.M. with the A.C.E.
                          • Produce a “one-number” census. No.                   program.
                          A.C.E.

                          • Measure census coverage.              No.            A.C.E. results were not used because
                                                                                 of uncertainties surrounding the
                          • Generate data needed for              No.            accuracy of the results and the inability
                            redistricting and federal program                    to resolve them in time to meet legally
                            purposes statistical methods.                        mandated deadlines for releasing data.
                          Source: GAO.

                          Note: This table reflects GAO’s analysis of U.S. Census Bureau data.




The Bureau Canceled       In designing I.C.M., the Bureau’s goal was to produce a single, consolidated
I.C.M. in Response to a   count or “one-number” census and thus avoid the controversy of having
                          two sets of census results as occurred during the 1990 Census. Thus, as
Supreme Court Ruling
                          shown in table 1, the objectives of I.C.M. were to (1) measure census
                          coverage, (2) generate, using statistical sampling and estimation methods,
                          the detailed data required for apportionment, congressional redistricting,
                          and federal program purposes, and (3) produce a one-number census.




                          Page 7                                    GAO-03-287 Census Coverage Measurement Programs
The Bureau’s plans for I.C.M. emerged in response to the unsatisfactory
results of the 1990 Census. Although the 1990 headcount was, at that time,
the most costly in U.S. history, it produced data that were less accurate
than those from the 1980 Census. The disappointing outcome was due in
large part to the Bureau’s efforts to count housing units that did not mail
back their census questionnaires. The operation, known as nonresponse
follow-up, where enumerators visited and collected information from each
nonresponding housing unit, proved to be costly and error-prone when a
higher-than-expected workload and a shortage of enumerators caused the
operation to fall behind schedule. The final stages of nonresponse follow-
up were particularly problematic. Indeed, while enumerators finished 90
percent of the follow-up workload within 8 weeks (2 weeks behind
schedule), it took another 6 weeks to resolve the remaining 10 percent.
Moreover, in trying to complete the last portion of nonresponse follow-up
cases, the Bureau accepted less complete responses and information from
nonhousehold members such as neighbors, which may have reduced the
quality of the data.

In the years following the 1990 Census, Congress, the Bureau, several
organizations, and GAO, concluded that fundamental design changes were
needed to reduce census costs and improve the quality of the data. In
response, the Bureau reengineered a number of operations for the 2000
Census.

For example, to save time and reduce its nonresponse follow-up workload,
the Bureau planned to enumerate a sample of the last remaining portion of
nonresponse follow-up cases instead of visiting every nonresponding
household as it had done in previous censuses. To adjust for enumeration
errors, the Bureau developed I.C.M., which was intended to reconcile the
original census figures with data obtained from a separate, independent
count of a sample of 750,000 housing units using a statistical process called
Dual System Estimation. The Bureau believed that this approach offered
the best combination of reduced costs, improved accuracy expected at
various geographic levels, and operational feasibility.




Page 8                          GAO-03-287 Census Coverage Measurement Programs
                             However, concerned about the legality of the Bureau’s planned use of
                             sampling and estimation, members of Congress challenged the Bureau’s
                             use of I.C.M. in court. In January 1999, the Supreme Court ruled that the
                             Census Act4 prohibited the use of statistical sampling to generate
                             population data for reapportioning the House of Representatives.5



A.C.E. Did Not Meet Bureau   Following the Supreme Court ruling, the Bureau planned to produce
Objectives                   apportionment numbers using traditional census-taking methods, and
                             provide statistically adjusted numbers for nonapportionment uses of the
                             data such as congressional redistricting and allocating federal funds. The
                             Bureau initiated the A.C.E. program, which was designed to take a national
                             sample of approximately 300,000 housing units to evaluate coverage errors
                             among different population groups and statistically correct for them. Thus,
                             as shown in table 1, the Bureau’s objectives for A.C.E. were to (1) measure
                             how many people were missed in the census and how many were
                             erroneously included and (2) produce the detailed data required in time for
                             redistricting and federal program purposes.

                             However, while the Bureau generally conducted A.C.E. in accordance with
                             its plans,6 the Bureau later determined that the A.C.E. results did not
                             provide a reliable measure of census accuracy and could not be used to
                             adjust the nonapportionment census data.




                             4
                              13 U.S.C. 195.
                             5
                              Department of Commerce v. United States House of Representatives 525, U.S. 316 (1999).
                             6
                              See, for example, U.S. General Accounting Office, 2000 Census: Coverage Evaluation
                             Interviewing Overcame Challenges, but Further Research Needed, GAO-02-26 (Washington,
                             D.C.: Dec. 31, 2001), and 2000 Census: Coverage Evaluation Matching Implemented as
                             Planned, but Census Bureau Should Evaluate Lessons Learned, GAO-02-297 (Washington,
                             D.C.: Mar. 14, 2002).




                             Page 9                               GAO-03-287 Census Coverage Measurement Programs
                             The first decision against A.C.E. occurred in March 2001, when the Acting
                             Director of the Census Bureau recommended to the Secretary of
                             Commerce that the unadjusted census data be used for redistricting
                             purposes. He cited as a primary reason an apparent inconsistency between
                             the population growth over the prior decade, as implied by A.C.E. results,
                             and demographic analysis, which estimated the population using birth,
                             death, and other administrative records. The inconsistency raised the
                             possibility of an unidentified error in either the A.C.E. or census numbers.
                             He reported that the inconsistency could not be resolved prior to April 1,
                             2001, the legally mandated deadline for releasing redistricting data.7

                             The second decision against A.C.E. came in October 2001 when, based on a
                             large body of additional research, ESCAP decided against adjusting census
                             data for allocating federal aid and other purposes, because A.C.E. failed to
                             identify a significant number of people erroneously included in the census,
                             and other remaining uncertainties. According to Bureau officials, it might
                             be possible to use adjusted data to produce intercensal population
                             estimates for federal programs that require this information; however, the
                             Bureau would need to revise the A.C.E. results before any use of the data
                             could be considered.



The Bureau’s Experience in   Although I.C.M. and A.C.E. did not meet their formal objectives, they did
Implementing                 produce a body of important lessons learned. As the Bureau’s current
                             approach for the 2010 Census includes coverage measurement to assess
Coverage Measurement
                             the accuracy of the census (but not necessarily to adjust the numbers
Programs                     themselves), it will be important for the Bureau to consider these lessons
Highlights Important         as its planning efforts continue. The lessons include (1) developing a
Lessons Learned              coverage measurement methodology that is both technically and
                             operationally feasible, (2) determining the level of geography at which
                             coverage measurement is intended, (3) keeping stakeholders, particularly
                             Congress, informed of the Bureau’s plans, and (4) adequately testing the
                             eventual coverage measurement program.

                             1. A.C.E. demonstrated operational, but not technical feasibility.
                                According to Bureau officials, an important result of the A.C.E.
                                program was that it demonstrated, from an operational perspective
                                only, the feasibility of conducting a large independent field check on

                             7
                             March 1, 2001, Memorandum to Secretary Donald Evans from Acting Director William
                             Barron, Jr. (ESCAP I decision memo.)




                             Page 10                            GAO-03-287 Census Coverage Measurement Programs
    the quality of the census. The Bureau canvassed the entire A.C.E.
    sample area to develop an address list, collected census response data
    for persons living in the sample areas on census day, and conducted an
    operation to try and match A.C.E. respondents to census respondents,
    all independent of the regular census operations and within required
    time frames.

    Our separate reviews of two of these operations—interviewing
    respondents and matching A.C.E. and census data—while raising
    questions about the impact on final A.C.E. results due to apparently
    small operational deviations, also concluded that the Bureau
    implemented those two operations largely as planned.8

    Nevertheless, while the Bureau demonstrated that it could execute
    A.C.E. field operations using available resources within required time
    frames, as the Bureau has noted, feasibility also consists of a technical
    component—that is, whether the A.C.E. methodology would improve
    the accuracy of the census. Although the Bureau clearly stated in its
    justification for A.C.E. that the effort would make the census more
    accurate, as noted earlier, because of unresolved data discrepancies, its
    experience in 2000 proved otherwise. Moreover, according to the
    Bureau, because the A.C.E. was designed to correct a census with a net
    coverage error similar to that observed in previous censuses, the
    Bureau commented that applying the methodology to the historically
    low levels of net error observed in the 2000 Census represented a
    unique and unanticipated challenge for A.C.E. Thus, it will be important
    for the Bureau to refine its coverage measurement methodology to
    ensure that it is technically feasible.




8
 See GAO-02-26 and GAO-02-297.




Page 11                          GAO-03-287 Census Coverage Measurement Programs
2. The level of geography at which the Bureau can successfully measure
   coverage is unclear. Since the October 2001 decision to not rely on
   adjusted census data for nonapportionment and nonredistricting
   purposes, Bureau officials have told us that they now doubt whether
   census data can reliably be improved down to the level of geography
   for which A.C.E. was intended to improve the accuracy—the census
   tract level (neighborhoods that typically contain around 1,700 housing
   units and 4,000 people). The Bureau’s current position differs from that
   taken in 2000, when it reported to Congress that it expected accuracy at
   the tract level to be improved, on average, by A.C.E. statistically
   adjusting numbers at an even lower level of geography—the census
   block level.9 Uncertainty in the level of geography at which accuracy is
   to be measured or improved can affect the overall design of coverage
   measurement, as well as its technical feasibility. Therefore, it will be
   important for the Bureau to determine the level of geography at which
   it intends to measure accuracy as it decides the role and design of
   future coverage measurement programs.

3. Keeping stakeholders informed is essential. Throughout the 1990s,
   Congress and other stakeholders, including GAO, expressed concerns
   about the Bureau’s planned use of sampling and statistical estimation
   procedures to adjust the census. A key cause of this skepticism was the
   Bureau’s failure to provide sufficiently detailed data on the effects that
   I.C.M. would have at different levels of geographic detail. Information
   was also lacking on the various design alternatives being considered,
   their likely implications, and the basis for certain decisions. As a result,
   it was difficult for Congress and other stakeholders to support the
   Bureau’s coverage measurement initiatives.




9
 U.S. Census Bureau, “Report to Congress—The Plan for Census 2000,” 44-46 and Accuracy and
Coverage Evaluation—Statement on the Feasibility of Using Statistical Methods to Improve
the Accuracy of the Census 2000 (June 2000), 19, fn. 19.




Page 12                              GAO-03-287 Census Coverage Measurement Programs
                               For example, on September 24, 1996, the House Committee on
                               Government Reform and Oversight issued a report that criticized the
                               Bureau’s initiatives for sampling and statistical estimation. Among
                               other things, the Committee found that the Bureau had not clarified
                               issues of accuracy, particularly for small geographic areas, raised by the
                               sampling initiative. Congress’s perspective on the process was later
                               reflected in its enactment of legislation in 1997 that included provisions
                               requiring the Department of Commerce to provide Congress with
                               comprehensive information on its planned use of statistical estimation
                               within 30 days.10

                       4. Adequate testing of coverage measurement methodologies is critical.
                          Although the Bureau conducted a dress rehearsal for the census in
                          three locations across the country that was intended to demonstrate
                          the overall design of the census, the 1998 operation did not reveal the
                          problems that the Bureau encountered in dealing with the
                          discrepancies between the 2000 A.C.E. results and its benchmarks.
                          According to Bureau officials, this was partly because the sites were
                          not representative of the nation at large. Additionally, as a result of a
                          compromise between Congress and the administration to
                          simultaneously prepare for a nonsampling census, the I.C.M. was tested
                          at only two of the three dress rehearsal sites—an urban area and an
                          Indian reservation—but was not tested in a rural location as was
                          originally planned. An earlier test in 1995 was also not comprehensive
                          in that it did not test a sampling operation designed to help determine
                          whether nonresponse follow-up of the magnitude projected by the
                          Bureau’s current plan could be completed in time for the I.C.M to be
                          done on schedule.



I.C.M./A.C.E.          From fiscal year 1996 through fiscal year 2001, the Bureau obligated about
                       $207 million for I.C.M./A.C.E. activities.11 As shown in table 2, of that
Contractor Costs Are   $207 million, we identified about $22.3 million (11 percent) in obligated
Not Complete           amounts for contracts involving more than 170 vendors. These contracts
                       were primarily for technical advisory and assistant services, computer
                       systems support, and training.



                       10
                            Pub. L. 105-18, Title VIII, June 12, 1997.
                       11
                            At the time of our report, obligated costs after fiscal year 2001 were not final.




                       Page 13                                      GAO-03-287 Census Coverage Measurement Programs
Table 2: Census 2000 I.C.M./A.C.E. Contractor Costs for Fiscal Years 1996 through 2001 (Dollars in thousands)

Project description                   FY96                FY97       FY98            FY99            FY00           FY01                  Total
A.C.E. coverage management                0                  0            0              0         $1,458                0            $1,458
A.C.E. operations                         0                  0            0              0            249             ($2)                 247
I.C.M. collection                         0                  0       $201              $41          6,272             467                 6,981
Framework 3 total                         0                  0         201              41          7,979             465                 8,686
I.C.M. procedures and training            0               $249         594           1,065          1,766             814                 4,488
I.C.M. processing                         0                  0            0          2,412            287             574                 3,273
Framework 5 total                         0                249         594           3,477          2,053           1,388                 7,761
I.C.M. dress rehearsal                    0                  0         502              20               0            (10)                 512
I.C.M. special test                    $182                628            0              0               0               0                 810
I.C.M. coverage measurement               0                409       1,257           1,991           (130)            999                 4,526
Framework 6 total                       182               1,037      1,759           2,011           (130)            989                 5,848
Total                                  $182          $1,286        $2,554          $5,529          $9,902          $2,842            $22,295
                                           Source: GAO.

                                           Note: This table reflects GAO’s analysis of U.S. Census Bureau financial management reports.


                                           Although the Bureau tracked some costs of contracts for the I.C.M./A.C.E.
                                           programs, we found that the $22.3 million did not represent the complete
                                           contractor costs of the programs because of the following three factors.

                                           • First, the Bureau only tracked the contractor costs associated with
                                             conducting the I.C.M./A.C.E. programs, which covers the period from
                                             fiscal year 1997 through 2003. Although life cycle costs for the 2000
                                             Census cover a 13-year period from fiscal years 1991 through 2003,
                                             senior Bureau officials said that the I.C.M./A.C.E. program was not
                                             viable for implementation until fiscal year 1997. Therefore, the Bureau
                                             considered contractor costs from earlier years as part of its general
                                             research and development programs, and the Bureau did not assign
                                             unique project codes to identify I.C.M./A.C.E. programs and related
                                             costs in its financial management system.

                                           • Second, although $182,000 of fiscal year 1996 obligated contractor costs
                                             were identifiable in the Bureau’s financial management system as an
                                             I.C.M. special test, the Bureau did not consider these costs as part of the
                                             I.C.M./A.C.E. programs. Instead, these costs were considered general
                                             research and development. However, because the Bureau separately
                                             identified these costs as I.C.M. program contractor costs, we have




                                           Page 14                                  GAO-03-287 Census Coverage Measurement Programs
                           included the $182,000 as part of the I.C.M./A.C.E. program contractor
                           costs in this report.

                    • Finally, we were unable to identify the I.C.M./A.C.E. portions of costs
                      that were part of other programs. For example, in late fiscal year 2000
                      and after, the Bureau did not separate A.C.E. evaluations from its other
                      2000 Census evaluations in its financial management systems. Bureau
                      officials stated that the contracts for evaluations included overall 2000
                      Census and A.C.E. evaluations, and did not have a separate code
                      identifying A.C.E. costs.



Tracking Future     During the 2000 Census, the Bureau, its auditors, and GAO, found extensive
                    weaknesses in the Bureau’s financial management system, the components
Coverage            of which include hardware, software, and associated personnel. The
Measurement Costs   weaknesses included difficulties in providing reliable and timely financial
                    information to manage current government operations and preparing
                    financial statements and other reports. Together, they affected the
                    completeness, accuracy, and timeliness of data needed for informed
                    management decisions and effective oversight. In light of these
                    weaknesses, the Bureau’s ability to track future costs of coverage
                    measurement activities will largely depend on three factors.

                    • First, a sound financial management system is critical. As discussed in
                      our December 2001 report, the Bureau’s core financial management
                      system, CAMS, had persistent internal control weaknesses in fiscal year
                      2000.12 In its latest financial report, the Bureau indicated that these
                      weaknesses have continued through fiscal year 2001.13 The Bureau
                      expects to issue its fiscal year 2002 financial report shortly.

                    • Second, it would be important to set up project codes to capture
                      coverage measurement activities as early in the planning process as
                      possible. The Bureau did not set up a specific project code to identify
                      I.C.M. program costs until 1996 because, according to Bureau officials,



                    12
                     U.S. General Accounting Office, 2000 Census: Analysis of Fiscal Year 2000 Budget and
                    Internal Control Weaknesses at the U.S. Census Bureau, GAO-02-30 (Washington, D.C.:
                    Dec. 28, 2001).
                    13
                         U.S. Census Bureau, 2001 Financial Report (Suitland, Md.: May 2002).




                    Page 15                                 GAO-03-287 Census Coverage Measurement Programs
                 the I.C.M. program was not viable until 1997 and all costs up to this
                 point were considered general research.

              • Finally, it would be important for Bureau personnel to correctly charge
                the project codes established for the coverage measurement program
                activities. During the 2000 Census, for example, while the Bureau
                established a project code and a budget for the remote Alaska
                enumeration, the project costs were erroneously charged to and
                commingled with a project code for enumerating special populations. As
                a result, the actual costs for remote Alaska enumeration were reported
                by the Bureau’s financial management system as zero and are unknown,
                while enumerating special population costs are overstated.



Conclusions   The Bureau’s 2000 Census coverage measurement programs did not
              achieve their primary objectives of measuring the accuracy of the census
              and adjusting the results because of legal challenges, technical hurdles, and
              questionable data. However, beyond these formal objectives, there
              emerged several important lessons learned that Bureau managers should
              consider because current plans for the 2010 Census include coverage
              measurement. At the same time, it will also be important for the Bureau to
              be capable of fully tracking the money it spends on coverage measurement
              and other census activities so that Congress and other stakeholders can
              hold the Bureau accountable for achieving intended results.

              Although the Bureau has never used the results of its coverage
              measurement programs to adjust census numbers, we believe that an
              evaluation of the accuracy and completeness of the census is critical given
              the many uses of census data, the importance of identifying the magnitude
              and characteristics of any under- and overcounts, and the cost of the
              census overall. Less clear is whether the results of the coverage
              measurement should be used to adjust the census. Any Bureau decisions
              on this matter should involve close consultation with Congress and other
              stakeholders, and be based on detailed data and a convincing
              demonstration of the feasibility of the Bureau's proposed approach.
              Whatever the decision, it is imperative that it be made soon so that the
              Bureau can design appropriate procedures and concentrate on the business
              of counting the nation’s population. The longer the 2010 planning process
              proceeds without a firm decision on the role of coverage measurement, the
              greater the risk of wasted resources and disappointing results.




              Page 16                         GAO-03-287 Census Coverage Measurement Programs
Recommendations for   To help ensure that any future coverage measurement efforts achieve their
                      intended objectives and costs can be properly tracked, we recommend that
Executive Action      the Secretary of Commerce direct the Bureau to

                      • in conjunction with Congress and other stakeholders, come to a
                        decision soon on whether and how coverage measurement will be used
                        in the 2010 Census;

                      • consider incorporating lessons learned from its coverage measurement
                        experience during the 2000 Census, such as (1) demonstrating both the
                        operational and technical feasibility of its coverage measurement
                        methods, (2) determining the level of geography at which coverage can
                        be reliably measured, (3) keeping Congress and other stakeholders
                        informed of its plans, and (4) adequately testing coverage measurement
                        prior to full implementation; and

                      • ensure that the Bureau’s financial management systems can capture and
                        report program activities early in the decennial process and ensure that
                        project costs are monitored for accuracy and completeness.



Agency Comments and   The Secretary of Commerce forwarded written comments from the Census
                      Bureau on a draft of this report, which are reprinted in appendix I. The
Our Evaluation        Bureau agreed with our recommendations highlighting the steps that
                      should be followed in the development of a coverage measurement
                      methodology for the 2010 Census and acknowledged their importance.
                      However, the Bureau maintained that it followed most of these steps for
                      the 2000 Census including (1) keeping stakeholders, particularly Congress,
                      informed of the Bureau’s plans, (2) determining the level of geography at
                      which coverage measurement is intended, and (3) adequately testing
                      coverage measurement methodologies. The Bureau also maintained that
                      throughout the 1990s, it had an open and transparent process for
                      implementing the coverage measurement program, including the levels of
                      geography to which its results would be applied.




                      Page 17                        GAO-03-287 Census Coverage Measurement Programs
We disagree. As we stated in our report, the Bureau’s failure to provide
important information was a key cause of congressional skepticism over
the Bureau’s coverage measurement plans. In fact, Congress was so
concerned about the lack of comprehensive information on the Bureau’s
proposed approach that in July 1997, it passed a law that included
provisions requiring the Department of Commerce to provide detailed data
on the Bureau’s planned use of statistical estimation within 30 days.14 We
revised the report to include this, and provide other examples to further
support our position that the Bureau’s I.C.M. and A.C.E. planning and
development processes were less than fully open and transparent.

The Bureau also commented that each major component of the
I.C.M./A.C.E. program underwent “rigorous” testing in the middle of the
decade as well as during the dress rehearsal for the 2000 Census held in
1998. We believe this overstates what actually occurred. As we noted in the
report, the dress rehearsal failed to detect the problems that A.C.E.
encountered during the 2000 Census because the sites were not
representative of the nation. Additionally, because of an agreement
between Congress and the administration to simultaneously prepare for a
census that did not include sampling, the I.C.M. was only tested at two of
the three dress rehearsal sites—an urban area and an Indian reservation—
but was not tested in a rural location as was originally planned. We made
this and other revisions to strengthen our point.

Because the A.C.E. was designed to correct a census with a net coverage
error similar to that observed in previous censuses, the Bureau commented
that applying the methodology to the historically low levels of net error
observed in the 2000 Census represented a unique and unexpected
challenge for A.C.E. We revised the report to reflect this additional context.

The Bureau took exception to the way we presented our conclusions
concerning its ability to properly classify certain costs associated with the
development of the Bureau’s coverage measurement programs. The Bureau
noted that it decided not to separately track coverage measurement
development costs in 1994, because there was no internal or external
request for a separate cost accounting of the program.

Our report does not make interpretive conclusions or qualitative judgments
about which coverage measurement program costs the Bureau decided to


14
     Pub. L. 105-18, Title VIII, June 12, 1997.




Page 18                                      GAO-03-287 Census Coverage Measurement Programs
track. Instead, the report (1) points out that we could not identify all of the
contractor costs associated with the I.C.M./A.C.E. programs because of the
three factors described in the report, and (2) underscores the importance
of a sound financial management system for tracking, planning, and
development costs for the 2010 Census.


We are sending copies of this report to other interested congressional
committees, the Secretary of Commerce, and the Director of the U.S.
Census Bureau. Copies will be made available to others upon request. This
report will also be available at no charge on GAO’s home page at
http://www.gao.gov.

Please contact Patricia A. Dalton on (202) 512-6806 or by E-mail at
daltonp@gao.gov if you have any questions. Other key contributors to this
report were Robert Goldenkoff, Roger Stoltz, Carolyn Samuels, Cindy
Brown-Barnes, Ty Mitchell, and Linda Brigham.

Sincerely yours,




Patricia A. Dalton
Director
Strategic Issues




McCoy Williams
Director
Financial Management and Assurance




Page 19                          GAO-03-287 Census Coverage Measurement Programs
Appendix I

Comments from the Department of                                        Appendx
                                                                             ies




Commerce                                                                Append
                                                                             x
                                                                             Ii




             Page 20     GAO-03-287 Census Coverage Measurement Programs
Appendix I
Comments from the Department of
Commerce




Page 21                           GAO-03-287 Census Coverage Measurement Programs
Appendix I
Comments from the Department of
Commerce




Page 22                           GAO-03-287 Census Coverage Measurement Programs
Related GAO Products


             2000 Census: Complete Costs of Coverage Evaluation Programs Are Not
             Available. GAO-03-41. Washington, D.C.: October 31, 2002.

             2000 Census: Lessons Learned for Planning a More Cost-Effective 2010
             Census. GAO-03-40. Washington, D.C.: October 31, 2002.

             2000 Census: Refinements to Full Count Review Program Could Improve
             Future Data Quality. GAO-02-562. Washington, D.C.: July 3, 2002.

             2000 Census: Coverage Evaluation Matching Implemented as Planned,
             but Census Bureau Should Evaluate Lessons Learned. GAO-02-297.
             Washington, D.C.: March 14, 2002.

             2000 Census: Best Practices and Lessons Learned for More Cost-Effective
             Nonresponse Follow-up. GAO-02-196. Washington, D.C.: February 11, 2002.

             2000 Census: Coverage Evaluation Interviewing Overcame Challenges,
             but Further Research Needed. GAO-02-26. Washington, D.C.:
             December 31, 2001.

             2000 Census: Analysis of Fiscal Year 2000 Budget and Internal Control
             Weaknesses at the U.S. Census Bureau. GAO-02-30. Washington, D.C.:
             December 28, 2001.

             2000 Census: Significant Increase in Cost Per Housing Unit Compared
             to 1990 Census. GAO-02-31. Washington, D.C.: December 11, 2001.

             2000 Census: Better Productivity Data Needed for Future Planning and
             Budgeting. GAO-02-4. Washington, D.C.: October 4, 2001.

             2000 Census: Review of Partnership Program Highlights Best Practices
             for Future Operations. GAO-01-579. Washington, D.C.: August 20, 2001.

             Decennial Censuses: Historical Data on Enumerator Productivity Are
             Limited. GAO-01-208R. Washington, D.C.: January 5, 2001.

             2000 Census: Information on Short- and Long-Form Response Rates.
             GAO/GGD-00-127R. Washington, D.C.: June 7, 2000.




(450133)     Page 23                      GAO-03-287 Census Coverage Measurement Programs
GAO’s Mission            The General Accounting Office, the investigative arm of Congress, exists to
                         support Congress in meeting its constitutional responsibilities and to help improve
                         the performance and accountability of the federal government for the American
                         people. GAO examines the use of public funds; evaluates federal programs and
                         policies; and provides analyses, recommendations, and other assistance to help
                         Congress make informed oversight, policy, and funding decisions. GAO’s
                         commitment to good government is reflected in its core values of accountability,
                         integrity, and reliability.


Obtaining Copies of      The fastest and easiest way to obtain copies of GAO documents at no cost is
                         through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail this
                         list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A check
                         or money order should be made out to the Superintendent of Documents. GAO
                         also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single
                         address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice: (202) 512-6000
                                                TDD: (202) 512-2537
                                                Fax: (202) 512-6061


To Report Fraud,         Contact:
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470



Public Affairs           Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
                         U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548
United States                  Presorted Standard
General Accounting Office      Postage & Fees Paid
Washington, D.C. 20548-0001           GAO
                                 Permit No. GI00
Official Business
Penalty for Private Use $300

Address Service Requested