oversight

Federal Programs: Ethnographic Studies Can Inform Agencies' Actions

Published by the Government Accountability Office on 2003-03-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Staff Study




March 2003
             FEDERAL
             PROGRAMS
             Ethnographic Studies
             Can Inform Agencies’
             Actions




GAO-03-455
             a
Contents


Preface                                                                                 1


Appendix I   How Ethnographic Studies Can Inform Agencies’
             Actions                                                                    3
             Introduction                                                              3
             Results in Brief                                                          3
             Background                                                                4
             Scope and Methodology                                                     9
             Federal Agencies Employ Ethnography in a Variety of Ways                  9
             Cases Illustrate Ethnography’s Incorporation in Agency Programs          14
             Concluding Observations                                                  27


Tables
             Table 1: Agencies’ Illustrative Uses of Ethnography                       10
             Table 2: Ethnographic Study Designs and Uses                              28


Figures
             Figure 1: The Ethnographic Research Process                                6
             Figure 2: Ethnographic Studies and Recommendations on
                      Enumerating Populations Difficult to Count                       23




             Page i                                        GAO-03-455 Federal Ethnography
Abbreviations

CDC               Centers for Disease Control and Prevention
DOD               Department of Defense
DSTDP             Division of STD Prevention
EHS               Early Head Start
EPA               Environmental Protection Agency
HHS               Department of Health and Human Services
NMFS              National Marine Fisheries Service
NOAA              National Oceanic and Atmospheric Administration
NPS               National Park Service
ONDCP             Office of National Drug Control Policy
SIA               social impact assessment
STD               sexually transmitted disease
VISTA™            Values in Strategy Assessment
YATS              Youth Attitudes Tracking Study




This is a work of the U.S. Government and is not subject to copyright protection in the
United States. It may be reproduced and distributed in its entirety without further
permission from GAO. It may contain copyrighted graphics, images or other materials.
Permission from the copyright holder may be necessary should you wish to reproduce
copyrighted materials separately from GAO’s product.




Page ii                                                 GAO-03-455 Federal Ethnography
United States General Accounting Office
Washington, DC 20548




                                   Preface
                                   In this time of emphasis on performance and results, federal agencies and
                                   congressional committees can benefit from knowing the full range of
                                   social science methods that can help them improve the programs they
                                   oversee. Among the methods they might consider are those of
                                   ethnography, derived from anthropology. However, information about the
                                   past and present uses of ethnography to improve federal programs has not
                                   been systematically gathered or analyzed. Therefore, its potential for
                                   program improvement may be overlooked.

                                   Ethnography can fill gaps in what we know about a community whose
                                   beliefs and behavior affect how federal programs operate. This can be
                                   especially useful when such beliefs or behavior present barriers to a
                                   program’s objectives. Ethnography helps build knowledge of a community
                                   by observing its members and by interviewing them in their natural setting.
                                   Although many people associate ethnography with lengthy anthropological
                                   research aimed at cultures remote from our own, it can be used to inform
                                   public programs and has a long history of application in the federal
                                   government.

                                   Our aim in this study has been to describe how federal agencies have used
                                   the results of ethnographic studies to improve agency programs, policies,
                                   and procedures. We have done this by examining the range and scope of
                                   the use of ethnography in the federal government. In working through a
                                   network of anthropologists in the federal government and other social
                                   science networks, among our other resources, we constructed a list of
                                   examples of the use of ethnography that were established, not merely one-
                                   time or grant-funded, elements in agency programs. From this list, we have
                                   been able to present case studies that illustrate different agencies whose
                                   application of ethnography varies in purpose, method, and the use of
                                   results. We believe that this study not only provides examples of how the
                                   federal government uses ethnography but also suggests some means of
                                   expanding and improving its use of ethnography and its results. We hope
                                   that it will provide a resource to agencies that face site-specific or broader
                                   issues involving communities or populations important to program
                                   success.



                                   Page 1                                          GAO-03-455 Federal Ethnography
Within the body of our text, we offer original visual presentations that
define the ethnographic research process and that suggest some
relationships between ethnographic studies and ways of enumerating
populations that are difficult to count. Readers will also find tables
summarizing our cases of the agencies’ uses of ethnography.

Copies of this study are available on request. For additional information,
please contact me at (202) 512-8430. Key contributors to this project
included Gail MacColl, Emily Jackson, and Tahra Edwards.




Donna Heivilin
Director, Applied Research and Methods




Page 2                                         GAO-03-455 Federal Ethnography
                   Appendix I: How Ethnographic Studies Can
Appendix I: How Ethnographic Studies Can
                   Inform Agencies’ Actions



Inform Agencies’ Actions

                   Gaps in information about and understanding of communities or
Introduction       populations whose beliefs and behavior affect how a federal program
                   operates can be obstacles to a program’s objectives. Those gaps are
                   sometimes filled by using ethnography, a set of social science methods
                   designed to build knowledge by observation and in-depth interviewing of a
                   community’s members. However, information about how federal agencies
                   have used ethnography has not been systematically gathered and analyzed,
                   and ethnography’s potential utility for evaluating programs and improving
                   them in a range of government settings has not been examined. Thus,
                   agencies and congressional committees have little help in ascertaining
                   when ethnographic methods might be useful for the programs they are
                   responsible for.

                   We undertook this study of federal agencies’ use of ethnographic methods
                   as a research and development project under our basic legislative
                   authority to undertake work in support of the Congress and of our
                   performance goal to improve the quality of evaluative information. Our
                   objectives were to (1) examine the range and scope of the use of
                   ethnography in the federal government and (2) illustrate how federal
                   agencies have used the results of ethnographic studies to improve agency
                   programs, policies, and procedures.


                   We identified program offices in 10 federal departments and agencies that
Results in Brief   have used ethnographic methods. They ranged from the National Park
                   Service (NPS), which has used ethnography to inform park planning and
                   interpretive programs, to the Bureau of the Census, which has used it to
                   conduct alternative enumerations of hard-to-count populations. In each
                   office, ethnography was used to gain a better understanding of the
                   sociocultural life of a group whose beliefs and behavior were important to
                   a federal program. The character and intensity of ethnographic studies and
                   the communities studied varied. For example, the Office of National Drug
                   Control Policy (ONDCP) uses ethnographers as consultants in its ongoing
                   monitoring of trends in street-level drug use. In contrast, the
                   Environmental Protection Agency (EPA) supported community-based
                   efforts to protect local drinking water by providing for ethnographers to
                   help local leaders understand cultural and social motivations and
                   obstacles to community involvement in those efforts.

                   Our case studies illustrate how four agencies used ethnographic methods
                   to meet statutory requirements and help meet performance objectives. The
                   National Marine Fisheries Service (NMFS) uses ethnography in conducting
                   statutorily required assessments of the impact of proposed fisheries’


                   Page 3                                       GAO-03-455 Federal Ethnography
             Appendix I: How Ethnographic Studies Can
             Inform Agencies’ Actions




             management plans on fishing communities. NMFS has also used
             ethnographies to help identify ways to mitigate the plans’ potentially
             negative effects. Rapid-assessment ethnographic procedures have helped
             Centers for Disease Control and Prevention (CDC) staff identify factors
             that contribute to outbreaks of sexually transmitted disease (STD) and
             suggest preventive actions to local health officials. The Bureau of the
             Census has used ethnographic studies to understand why certain groups
             have been undercounted and to propose improved methods of
             enumeration. To target recruiting efforts more effectively to young people,
             the Department of Defense (DOD) used ethnographic surveys to obtain in-
             depth information about the propensity to join the military from youths
             who had participated in an ongoing structured survey and their parents.

             In each case, a program’s operation or outcomes depended in some way
             on the actions of a definable cultural community. Overall, ethnographic
             methods were used to obtain new information about and understanding of
             the communities and the problem at hand through interaction with
             community members. In some instances, findings were applied only to the
             particular program’s local community. But the case study agencies also
             sought to capture ethnographic information across study sites in order to
             support comparisons, reliably identify recurring themes, or facilitate the
             integration of ethnographic with economic or other quantitative data.


             Ethnography is a social science method developed within cultural
Background   anthropology for studying communities in natural settings. Although
             ethnography is commonly associated with lengthy research aimed at
             understanding cultures remote from our own, it can also be used to inform
             the design, implementation, and evaluation of public programs.
             Ethnography has a long history of application in the federal government.
             For example, in 1852, the ethnographic work of an anthropologist
             commissioned by the Congress to report on the circumstances and
             prospects of Indian tribes in the United States gave background and
             direction to Indian policy. During the era of the New Deal, anthropologists
             in the U.S. Department of Agriculture examined the problems of rural
             poverty and the relationship of farming to community viability.
             Anthropologists during World War II established institutes at universities
             across the country to teach foreign service officers, military personnel,
             and others regional history, language, culture, society, and politics
             relevant to national defense and U.S. participation in global affairs.

             In this study, we focus on how federal agencies can use ethnographic
             techniques to understand and address issues or problems important to a


             Page 4                                        GAO-03-455 Federal Ethnography
                        Appendix I: How Ethnographic Studies Can
                        Inform Agencies’ Actions




                        program, such as the relationship between natural resources and
                        communities or between community beliefs and behaviors and program
                        activities. In the remainder of this section, we describe some of the
                        defining features of ethnography as a method of study.


Ethnographic Methods    Ethnographic methods are exploratory. They are appropriate for studying
Explore Behavior in a   issues or problems that are not readily amenable to traditional quantitative
Social Setting          or experimental methods alone and in which it is important to discover
                        what the participants do and why they do it from their own perspective.1
                        Ethnographic techniques are used to

                        •   define an issue or problem when it is not clear, when it is complex, or
                            when it is embedded in multiple systems or sectors;

                        •   identify the range of the problem’s settings and the participants,
                            sectors, or stakeholders in those settings who are not known or who
                            have not been identified;

                        •   explore the factors associated with the issue or problem in order to
                            understand and address them or to identify them when they are not
                            known;

                        •   describe unexpected or unanticipated outcomes; and

                        •   design measures that match the characteristics of the target
                            population, clients, or community participants when existing measures
                            are not a good fit.

                        Ethnographic studies follow the steps illustrated in figure 1. In the
                        preparation stage, the study team assembles existing information about
                        the local community to be studied. The research often begins with an
                        initial question about a situation—for example, “What accounts for the
                        undercount of urban minorities during the decennial census?” The
                        researcher uses prior knowledge about a setting to create a formative
                        theory or explanation of the situation.




                        1
                         For a comprehensive discussion of ethnographic methods, see Jean J. Schensul and
                        Margaret D. Le Compte (eds.), Ethnographer’s Toolkit (Walnut Creek, Calif.: AltaMira
                        Press, 1999).




                        Page 5                                                 GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




Figure 1: The Ethnographic Research Process

    Preparation, initial                                                                 Final
    framework                    Field data collection, recording, analytic reflection   coding        Final analysis




                                                         Mental notes
      Existing
      information            Human interaction
                             and observation in a local setting
                                                                    Scratch
                                                                    notes                 Organizing
                                                                                          data, more
                                                                                          analysis

                                                                        Field notes,
                             Initial theory                             descriptions,
                                                                        and analysis
                                      Revised
                                      theory


      Prior
      knowledge




    Source: GAO and Art Explosion.




Then field data collection, the next stage, begins. Ethnography is
distinguished by the collection of data by means of human observation and
interaction in a local setting, with the researcher as the primary data
collection tool.

Key exploratory ethnographic data collection techniques include

•      Exploratory or participant observation, which requires the
       ethnographer to be present at, involved in, and recording the daily
       activities in the field setting. It is the starting point for ethnographic
       research.

•      In-depth, open-ended interviewing, which explores a topic in detail
       in order to deepen the interviewer’s knowledge of an area about which
       little is known. The interviewer is open to all relevant responses and
       has flexibility to cover new topics as they arise.




Page 6                                                                        GAO-03-455 Federal Ethnography
                              Appendix I: How Ethnographic Studies Can
                              Inform Agencies’ Actions




                              •   Semi-structured interviewing, which is more focused than open-
                                  ended interviewing and may be administered to a representative
                                  sample of respondents. It is based on an interview guide of
                                  predetermined questions but seeks open-ended answers and allows the
                                  researcher to expand questions through probes.2

                              Fieldwork varies in depth and duration. In studies of an entire culture, the
                              ethnographer may be fully immersed in the research setting, living in the
                              setting and participating in all possible daily activities for a number of
                              years. Ethnographic studies that focus on a particular dimension or aspect
                              of culture often involve shorter periods. For example, they may use
                              techniques such as a rapid assessment procedure to gather in-depth
                              information on a particular problem in a short time.

                              In circumstances where it is inappropriate to take notes openly,
                              ethnographers may make mental notes that they keep in memory until it is
                              possible to write them down. They make scratch notes, or jottings and
                              scribblings during events in the field, or immediately after events, often
                              written in shorthand or code; these help the ethnographer remember their
                              mental notes. Ethnographers later turn their mental notes and scratch
                              notes into descriptions, developed in turn into field notes, which make a
                              custom, belief, or practice comprehensible not only to the ethnographer
                              but also to outsiders. In other circumstances, where information collected
                              is of legal significance, federal ethnographers may record interviews
                              openly, making sure that respondents know that their answers are being
                              recorded.


Analysis and Interpretation   In an ethnographic study, analysis and interpretation proceed from the
Proceed Throughout an         moment the researcher enters the field.3 As depicted in figure 1, the
Ethnographic Study            researcher’s initial theory is continually modified as new information is
                              obtained. Field notes include reflection, preliminary analysis, initial
                              interpretations, and new questions to be explored and tested in future
                              observations or interviews. They are organized around the basic


                              2
                               In addition to these general techniques, ethnography has specialized techniques for
                              mapping social networks or spatial data, audiovisual techniques, and focused group
                              interviewing. Schensul and LeCompte (eds.), vols. 3 and 4, discuss these methods.
                              3
                               Analysis refers to the process of coding, counting, tallying, and summarizing so as to
                              reduce field notes to manageable form and to reveal patterns and themes. Analysis enables
                              the ethnographer to tell a story about the group that is the focus of research. Interpretation
                              permits the ethnographer to tell readers what the story means.




                              Page 7                                                    GAO-03-455 Federal Ethnography
                           Appendix I: How Ethnographic Studies Can
                           Inform Agencies’ Actions




                           conceptual frames or questions that structured the study in the beginning,
                           and they become increasingly focused as research progresses.

                           Analysis out of the field begins with final coding of text data—that is,
                           organizing the data in terms of a framework with which the researcher can
                           support the analysis and reach conclusions. The researcher decides which
                           data should be coded and devises a strategy for separating them into
                           conceptual categories. After the data are coded, researchers can begin to
                           examine collections of codes, through a variety of systematic qualitative
                           analysis techniques, to see how they are related to one another, and
                           organize related items by their patterns. Qualitative data can also be
                           integrated with or compared to results from surveys or other sources.

                           Reporting results from ethnographic research commonly involves
                           constructing an analytically informed narrative portrait of what is being
                           studied. The narrative may include vignettes or descriptions intended to
                           convey typical events or situations, as well as contingency tables and other
                           graphic or numerical displays.


Ethnography Overlaps but   Field observation and exploratory interviews are not exclusive to
Is Distinct from Other     ethnography: Other social sciences draw on them as well. And
Social Sciences            ethnographic work may include measurement and analysis methods
                           common to other social sciences. For example, it may include a structured
                           survey administered to a representative sample of respondents that can be
                           used to measure relationships among the variables that emerged from
                           exploratory work. The standards of quality that apply to these methods
                           apply in ethnographic studies, as they would in any social science.

                           However, two important features distinguish the ethnographic approach
                           from other social sciences. First, ethnography seeks to understand
                           culturally based behaviors and beliefs from the perspective of a
                           community’s members and to use local perspectives as the foundation for
                           building testable theories. Second, the researcher is the primary tool for
                           data collection, which takes place under conditions that the ethnographer
                           cannot control. The second feature raises potential threats to validity and
                           reliability, but these can be minimized by careful procedures that are




                           Page 8                                         GAO-03-455 Federal Ethnography
                       Appendix I: How Ethnographic Studies Can
                       Inform Agencies’ Actions




                       outlined in the professional ethnographic literature and are emphasized in
                       ethnographers’ training.4


                       To ascertain the range and scope of the federal agencies’ use of
Scope and              ethnography, we solicited information through a network of
Methodology            anthropologists in the federal government and other social science
                       networks and newsletters, conducted literature reviews, and searched
                       agency Internet sites. Given the limitations of our initial search, some
                       federal agency uses may not have come to our attention, and our list of
                       agencies that have used ethnography should not be interpreted as
                       complete.

                       From initial information from our search and follow-up conversations with
                       individuals who identified agency uses, we constructed a list of examples
                       of the use of ethnography that were an established element in a program
                       (not merely one-time or grant-funded studies) and that were sufficiently
                       mature to have generated results. For more detailed study, we then chose
                       four examples that illustrate different agencies whose use of ethnography
                       varied in purpose, method, and use of results. We obtained additional
                       information about these programs from agency documents, external
                       reports that discussed the ethnographic studies, and interviews with
                       knowledgeable officials. Our four case studies do not constitute a
                       complete set or representative sample of established agency uses. They
                       are intended only to be illustrative.

                       We requested comments on a draft of this report from the Department of
                       Commerce and the Department of Defense and the National Center for
                       HIV, STD, and TB Prevention. DOD and Commerce provided technical
                       comments that we incorporated where appropriate throughout the report.



                       Table 1 shows program offices in 10 federal departments or agencies that
Federal Agencies       we found employed ethnographic methods in the past 15 years. They
Employ Ethnography     include NPS’s work to ensure that park planning and interpretive
                       programs are culturally informed and the Bureau of the Census’s
in a Variety of Ways   enumerating in alternative ways populations that are difficult to count. In



                       4
                        Schensul and Le Compte (eds.), Ethnographer’s Toolkit, vols. 2–4, address these issues in
                       connection with a variety of methods.




                       Page 9                                                  GAO-03-455 Federal Ethnography
                                          Appendix I: How Ethnographic Studies Can
                                          Inform Agencies’ Actions




                                          each case, we found that ethnography was used to understand better a
                                          group’s sociocultural life with respect to an important federal program
                                          issue.

Table 1: Agencies’ Illustrative Uses of Ethnography

 Sponsoring agency                                                Population studied                  Purpose
 Department of Commerce         Bureau of the Census              Urban and rural, hard-to-count      Examine factors contributing to
                                                                  populations                         undercounts and conduct an
                                                                                                      alternative enumeration in the
                                                                                                      sites studied

                                National Marine Fisheries         Members of fishing                  Gauge the potential social and
                                Service                           communities                         cultural consequences of
                                                                                                      alternative fishery management
                                                                                                      actions or policies on local
                                                                                                      communities
 Department of Defense          Defense Human Resources           Youths of military enrollment       Understand factors affecting
                                Activity                          age                                 youths’ propensity to join the
                                                                                                      military and strengthen
                                                                                                      recruitment targeting them

                                Naval Health Research Center      Military winter-over personnel in Understand and help improve
                                                                  Antarctica                        health and performance of
                                                                                                    personnel under conditions of
                                                                                                    prolonged isolation in an
                                                                                                    extreme environment
 Department of Health and       Administration for Children and   Early Head Start children and     Examine how children and their
 Human Services                 Families                          families                          families experienced a
                                                                                                    Montessori preschool program
                                                                                                    and help interpret patterns in
                                                                                                    quantitative data

                                Centers for Disease Control       Local populations at risk for or    Develop intervention strategies
                                and Prevention                    experiencing higher rates of        specific to areas in which
                                                                  STDs                                increases in STDs occur to
                                                                                                      prevent further transmission

                                Centers for Medicare and          Nurses and managers in long-        Explore management practices
                                Medicaid Services                 term care facilities                and other factors that affect
                                                                                                      nursing staff recruitment and
                                                                                                      retention

                                National Institute on Drug        Populations of drug users           Obtain detailed information
                                Abuse                                                                 about patterns of drug use and
                                                                                                      emerging trends
 Department of Housing and                                        Populations that have               Gain better understanding of
 Urban Development                                                traditionally lacked access to      barriers to homeownership for
                                                                  home mortgage lending               these groups and inform policy




                                          Page 10                                                    GAO-03-455 Federal Ethnography
                                            Appendix I: How Ethnographic Studies Can
                                            Inform Agencies’ Actions




 Sponsoring agency                                                Population studied               Purpose
 Department of the Interior        National Park Service          Members of present-day tribes    Support policy making, program
                                                                  and communities associated       planning, and management and
                                                                  with national parks              ensure that park site
                                                                                                   interpretations are culturally
                                                                                                   informed
 Department of Justice             Office of Justice Programs     Drug-users, dealers, and gang Build knowledge of criminal
                                                                  members and other populations behavior that can be applied to
                                                                  involved in street crime         the operations of the justice
                                                                                                   system
 Department of Labor               Employment and Training        Communities targeted by          Assess the neighborhoods’
                                   Administration                 Labor’s youth opportunity        sense of well being before,
                                                                  programs                         during, and after the Youth
                                                                                                   Opportunity Area
                                                                                                   Demonstration in order to
                                                                                                   evaluate the program’s overall
                                                                                                   effectiveness
 Environmental Protection          Office of Ground Water and     Neighborhoods involved in        Provide details about
 Agency                            Drinking Water                 community-based                  community values and
                                                                  environmental protection efforts perspectives that relate to
                                                                                                   community-based
                                                                                                   environmental behavior
 Office of National Drug Control                                  Drug-using populations           Provide information that can
 Policy                                                                                            alert policy makers to short-
                                                                                                   term changes or newly
                                                                                                   emerging problems on specific
                                                                                                   drugs, drug users, or drug
                                                                                                   sellers
 U.S. Agency for International                                    Overseas communities targeted Provide a more detailed picture
 Development                                                      for USAID programs               of the local context in which a
                                                                                                   project will be carried out in
                                                                                                   order to tailor it better to the
                                                                                                   needs of the target population
Source: Agency documents.

                                            Before turning to our more extended examples, we summarize a few that
                                            illustrate the range of the agencies’ use of ethnography. The National Park
                                            Service established its Applied Ethnography Program to systematically
                                            conduct, contract for, and share the results of studies in cultural
                                            anthropology with park planners, managers, and other decision makers.
                                            The program uses ethnographic studies to identify diverse present-day
                                            tribes and communities and to ascertain the cultural meanings of park
                                            resources to their members. The results of the studies are used in
                                            assessing the impact of planned actions on the cultural and natural
                                            environment, as required pursuant to the National Environmental Policy
                                            Act, and on sacred places, pursuant to the American Indian Religious
                                            Freedom Act (and Executive Order 13007 on Sacred Sites). In addition,
                                            ethnographic studies help NPS ensure that diverse local views and
                                            community concerns are considered in park planning and that NPS’s site
                                            presentations to visitors are culturally informed. This can be particularly



                                            Page 11                                               GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




helpful in interpreting sites that have a divisive past, such as massacre
sites, where American Indians lost their lives, and plantation parks, which
are characterized by strong, often divergent views of slavery among
descendants of enslaved people and their owners.

EPA, recognizing the importance of human and social needs and local
community actions in ecosystem protection, has drawn on ethnography to
support such community actions. Staff in EPA’s Office of Wetlands,
Oceans, and Watersheds created a guide that incorporates ethnographic
approaches in outlining procedures for identifying community cultural
values, beliefs, and behaviors related to the environment.5 The guide is
intended for training people, organizations, and institutions in watershed
protection and public health professionals as well as federal, tribal, state,
and local agencies seeking technical skills for improving stakeholder
involvement. EPA has also made ethnographic consultants available. In
1996, EPA’s Office of Ground Water and Drinking Water set up a 5-year
cooperative agreement with the Society for Applied Anthropology to assist
community-based efforts to protect local drinking water supplies and to
understand the cultural and social motivations and obstacles to
community involvement at the local level. For example, an anthropologist
helped the Lower Elwha Klallam Tribe explore how its traditional cultural
emphasis on loose local control of natural resources may affect its ability
to adopt source water protection programs and helped tribal members see
the need for protective management in light of the reservation’s growing
population.

ONDCP uses ethnographic expertise to alert policy makers to short-term
changes and newly emerging problems concerning specific drugs, drug
users, and drug sellers. An ONDCP quarterly publication, Pulse Check,
uses reports from telephone interviews with ethnographers and
epidemiologists who work in drug research to gather information about
trends in use, areas where drugs are sold, availability, quality, and pricing.
Pulse Check provides a quick sense of what is happening with regard to
drug abuse across the nation.

The Department of Justice has turned to ethnographic research to learn
about street crime and gang activity. For example, the National Institute of
Justice’s 1999 annual conference on criminal justice research and



5
 EPA, “Community, Culture, and the Environment: A Guide to Understanding a Sense of
Place,” Washington, D.C., November 2002.




Page 12                                              GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




evaluation featured work by ethnographers who had studied crime and
drugs from the street level.6 In one study, ethnography was used to obtain
quantitative information—numerical data on the economic practices of a
street gang (wage and nonwage expenditures)—through key informants
and participant observation. Ethnography was helpful in gathering two
data sets that may assist in the policy arena: (1) the ethnographer gained
access to financial records a single gang maintained for a 4-year period
that contained data on the price and quantity of drugs sold, other sources
of revenue, and gang expenditures for wages, weapons, funerals, and other
items and (2) ethnographers and macroeconomists longitudinally tracked
nearly a dozen gangs in a large city, producing a taxonomy of the various
organizational structures in which drug trafficking took place.

The Administration for Children and Families of the Department of Health
and Human Services (HHS) used ethnographic techniques in two national
Early Head Start (EHS) evaluation sites to illuminate ways in which the
families EHS served accepted or rejected the program’s Montessori
intervention. The national EHS evaluation follows a traditional random-
assignment research design, with quantitative measures of process and
outcome. Several sites, however, included anthropological work as part of
their local research to tell the story of program implementation more fully
and to document the sociocultural contexts in which programs operated.
An ethnographer observed the classroom regularly and visited selected
families in their homes to discuss their thoughts about the Montessori EHS
intervention in more detail. The preliminary results from this ethnographic
research have emphasized that contrary to what may have been believed
about Montessori before the program’s experience, low-income parents
appreciated and valued the changes they saw in their children.

The Employment and Training Administration in the Department of Labor
included an ethnographic community assessment in an evaluation of the
1996 Youth Opportunity Area (Kulick) Initiative for out-of-school youths.
Findings from the ethnographic component, along with a process
evaluation and survey data, will be integrated into an overall evaluation of
the effectiveness of the Kulick project. Ethnographic methods in this
context are used to assess the neighborhoods’ sense of well-being before,
during, and after the demonstration.



6
 The conference is reported in Looking at Crime from the Street Level: Plenary Papers of
the 1999 Conference on Criminal Justice Research and Evaluation—Enhancing Policy
and Practice through Research, vol. I (Washington, D.C.: Department of Justice, 1999).




Page 13                                                GAO-03-455 Federal Ethnography
                          Appendix I: How Ethnographic Studies Can
                          Inform Agencies’ Actions




                          The examples below illustrate how a regulatory agency, a preventive
Cases Illustrate          health services agency, a statistical agency, and a defense agency
Ethnography’s             incorporated ethnographic methods to shed light on program-relevant
                          aspects of cultural communities critical to program success. The
Incorporation in          ethnographic work represented immersion in a community for an
Agency Programs           extended period, rapid assessment, and cognitive interviewing by external
                          and in-house experts. The cases include both immediate and long-term
                          results of study findings.


NMFS Uses Ethnographic    NMFS, a unit within the National Oceanic and Atmospheric Administration
Techniques in Assessing   (NOAA) in the Department of Commerce, is responsible for the science-
the Effects of Fishery    based conservation and management of the nation’s living marine
                          resources and their environment. In allocating fishing resources through
Management Plans          the fisheries management planning process, NMFS is required by statute to
                          consider the impact of its allocations on fishing communities.
                          Ethnographic studies have contributed to this decision making process.

                          NMFS is responsible for upholding the guidelines and regulations
                          promulgated under the Magnuson-Stevens Fishery Conservation and
                          Management Act.7 The act established eight Regional Fishery Management
                          Councils, giving them the responsibility of managing fisheries resources
                          for conservation purposes and for allocating these resources fairly among
                          various and competing users, such as commercial, charter, and
                          recreational fishers. To carry out this responsibility, the councils prepare
                          recommendations, known as fishery management plans, to the Secretary
                          of Commerce. The plans are sent first to the appropriate NMFS Regional
                          Office and then to NMFS’s Office of Sustainable Fisheries, where they are
                          checked for compliance with this act and other laws concerning marine
                          resources that require agencies to assess the social and economic effects
                          of proposed regulatory or policy changes.8 Fishery plans that pass the
                          review are sent to the Secretary of Commerce for approval. Once a plan is
                          approved, NMFS issues regulations to implement it.9



                          7
                          16 U.S.C. §§1853–63 (2000). The act gave this responsibility to the Secretary of Commerce,
                          who delegated it to NMFS.
                          8
                          Examples include the National Environmental Policy Act and Executive Order 12898 on
                          Environmental Justice.
                          9
                           U.S. General Accounting Office, Individual Fishing Quotas: Better Information Could
                          Improve Program Management, GAO-03-159 (Washington, D.C.: Dec. 11, 2002).




                          Page 14                                                 GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




The Magnuson-Stevens Fishery Conservation and Management Act
stipulates that conservation and management measures take into account
the importance of fishery resources to fishing communities in order to
provide for the communities’ sustained participation and, to the extent
practicable, minimize adverse economic impact on them.10 This is done by
conducting social impact assessments (SIA), which are like economic or
ecological impact assessments but focus on the human environment of
fisheries—on the effects of change in resource availability or fishing
practices on fishermen, communities, fishing-related businesses and
employment, families and other social institutions, social norms of
behavior, and cultural values. Social impact assessments provide a basis
for assessing the social and cultural consequences of alternative fishery
management actions or policies. NMFS considers them an essential
element in fisheries’ decision making.

Many such assessments used in fisheries plans are supported by
ethnographic studies, some performed by firms under contract to the
regional council, others by academics. In addition, a social anthropologist
on the NOAA staff reviews each proposed plan for compliance with the
assessment’s requirement. As outlined in a recent NOAA guidance
document, conducting an SIA to fulfill the standards of the Magnuson-
Stevens Fishery Conservation and Management Act involves

•     identifying communities substantially dependent on fishing within the
      area that is affected,

•     establishing a baseline profile of each community that covers a range
      of economic and sociocultural variables,

•     describing and analyzing community social factors,

•     identifying fisheries’ issues from the community members’ perspectives
      and collecting and analyzing information on them, and

•     assessing alternative management actions with respect to these issues
      and their likely effects on the community.

To illustrate, an SIA using a rapid ethnographic assessment method was
conducted with respect to a Multispecies Groundfish Fishery in New



10
    16 U.S.C. §1851(a)(8)(2000).




Page 15                                         GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




England and the Mid-Atlantic region.11 The ports studied were selected by
using a combination of information derived from field visits, licensing
data, telephone interviews in the local area, and consultation with national
and regional NMFS representatives. Both social and economic aspects of
the Multispecies Groundfish Fishery fleet were identified and described,
based on information from in-depth interviews, focus groups, and other
sources. The assessment report noted each port’s dependence on the
fishery and the role of available public programs, explained the fisheries
issues that fishing communities faced, and contrasted issues fishermen
identified with those of fishery managers. This rapid SIA emphasized the
importance of understanding the nature and extent of the Multispecies
Groundfish Fishery crisis and the unique characteristics and adaptive
strategies of its fisher families and communities.

NMFS officials cited instances in which ethnographic assessments had
provided key information for fisheries’ plans. For example, in 1990 a social
science consulting firm developed community profiles for the North
Pacific Fisheries Management Council that concerned the allocation of the
catch between various participants. It addressed the critical question of
how the fishing catch in the region should be allocated between the
mother ship fleet (floating processors), the factory trawler fleet, and the
onshore processors. According to an NMFS official, an economic impact
statement had also been performed in the area, but its data had stated that
there would be little change in the cost of fish to consumers if catch were
restricted for any one of these groups. However, the social data showed
that if the fisheries management plan did not properly allocate catch
between the three groups, there would be significant disruption in the
social and cultural lives of the fishers in Alaskan communities. There were
also some hidden economic costs, such as welfare payments to the fishers
who suffered because their catch allocation was reduced. According to the
NMFS official, the community profiles were incorporated into the plan and
served as the basis for the Inshore/Offshore Allocation Agreement in 1991.
They were also used as building blocks, in combination with the
preliminary results of the economic modeling group, for future reports
analyzing the potential social effects of different inshore–offshore
allocative regulations.

NMFS and the regions have in the past had few social scientists on staff
and have relied largely on sociocultural studies by outside experts, each


11
 Groundfish are fishes that swim near the bottom, such as cod, haddock, and flounder.




Page 16                                                GAO-03-455 Federal Ethnography
                           Appendix I: How Ethnographic Studies Can
                           Inform Agencies’ Actions




                           with its own design. While individually designed studies can provide useful
                           insight into local or regional conditions in a given year, study-to-study
                           differences in specific variables and analyses make it difficult to analyze
                           study results over time. Noting an increase in legal challenges to fishery
                           plans, NMFS has recently taken steps to build its capacity to assess and
                           analyze social, cultural, and economic issues and impacts related to
                           fisheries’ management actions. This includes hiring additional social
                           scientists as well as revising existing NMFS operational guidelines and
                           creating new guidelines and manuals to help social science professionals
                           and fisheries managers research and write the numerous analyses for each
                           management action.

                           NMFS officials are also working on creating a longitudinal database for
                           sociocultural data. The Sociocultural Practitioners Manual recommends a
                           basic set of demographic, social structural, cultural, and socioeconomic
                           data elements to be collected for every fishery, fishing community, and
                           fishing port in order to create community profiles that will support
                           comparative analysis—over time, across fisheries, and nationally.
                           Activities to develop a common database involve economists and experts
                           in information technology as well as social scientists knowledgeable about
                           ethnography. These activities are coordinated with a NOAA science quality
                           assurance project and the development of agency guidelines under the
                           Information Quality Law, which covers all agency-sponsored documents
                           and presentations, including fisheries management plans signed by the
                           agency’s secretary.12


CDC’s Rapid Ethnographic   CDC is the nation’s focus for developing and applying disease prevention
Assessments Help           and control, environmental health, and health promotion and education
Communities Respond to     activities designed to improve the health of the people of the United
                           States. One of today’s most critical public health challenges is controlling
Outbreaks of Sexually      and preventing STDs, not only because they are the most frequently
Transmitted Disease        reported communicable diseases in the country but also because they
                           cause largely preventable, severe, and costly complications in the nation’s
                           most vulnerable populations.13 Toward this end, CDC’s Division of STD
                           Prevention (DSTDP) employs the rapid assessment procedure, an


                           12
                            Treasury and General Government Appropriation Act, 2001, §515, Pub. L. 106-554, H.R.
                           5658.
                           13
                            STDs have severe consequences for women and infants (especially ethnic and racial
                           minority populations).




                           Page 17                                               GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




ethnographic technique, to help state and local health departments
develop STD prevention strategies. Often local health departments ask for
assistance, but CDC also initiates the procedure, based on data that show
increases in STD infection. In offering this technical assistance, DSTDP is
faced with two challenges: (1) to identify the social and environmental
causes of a disease outbreak in a local area and (2) to develop intervention
strategies specific to the area in which the increase in STD is occurring to
prevent further transmission of disease.

The rapid assessment procedure is concerned specifically with collecting
information about beliefs and perceptions about health, the prevention
and treatment of illness, and community members’ interaction with local
health services. A team, usually consisting of four CDC staff members,
including the senior research anthropologist in DSTDP plus others trained
in public health, sociology, or psychology, typically comes from DSTDP
but may include individuals from the HIV/AIDS unit, if appropriate. The
local health department that is the “customer” for the procedure
sometimes contributes employees to assist in the effort.

Rapid assessment procedures have been conducted both during and after
an STD outbreak. To prepare for a site visit, the team typically spends
about 2 weeks collecting background information about the community
through the Internet and other secondary data sources. Team members
also talk with health department officials to identify key informants, and
from this initial conversation, a list of people to talk with and places to go
to “snowballs.”

On arriving at the site, the team meets with local health officials to learn
more about their needs in addressing the increase in STDs, learn their
perceptions of existing problems, and identify areas and issues about
which additional information is needed. Next, they enter the field to gather
information from community members, including teachers; health
personnel; community, youth, and religious leaders; bar owners; taxi
drivers; prostitutes; and other people prominent in the target population.
The team uses a rapid assessment procedure fieldwork guide, based on a
widely available model.14 Many questions in the guide are open-ended
regarding beliefs and perceptions of STDs in the community. For example,



14
 Susan Scrimshaw and Elena Hurtado, Rapid Assessment Procedures for Nutrition and
Primary Health Care: Anthropological Approaches to Improving Programme
Effectiveness (New York: United Nations University, 1987).




Page 18                                            GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




one question from the Los Angeles, California, ethnographic interview
guide is, “What do you think life is like for HIV-positive men living in Los
Angeles?” Another question is, “How would you describe the impact of
STDs among communities of men having sex with men?” Broad, open-
ended questions give the persons being interviewed the freedom to discuss
a range of issues in the local context that they feel are important. The
interview guide also includes more specific questions, called probe
questions, that interviewers should ask only if the information has not
already been addressed in earlier responses. An example is, “What about
the idea that young men who have sex with men don’t know people who
have HIV or who have died from AIDS and are more willing partners for
that reason?”

In addition to conducting interviews, the team looks for physical
community characteristics that facilitate the transmission of disease—
such as the presence of sex clubs and other establishments that are fertile
ground for STDs—and for health program practices that are barriers to
seeking care. Where relevant, team members may examine law
enforcement or other institutional practices that impede disease
prevention.

The rapid assessment procedure team typically spends about a week in the
field recording notes, following the procedures outlined in the background
section of this report. At the end of the study, team members report
findings to site officials and collaborate with the program officers and
other stakeholders to develop recommendations. Recommendations focus
on interventions at the structural or system level, such as changing
procedures within the local health department, and are strictly advisory.
However, DSTDP staff indicated that because DSTDP and the local
officials develop the interventions collaboratively, with the intention of
stopping the increase in STDs, local health officials are highly motivated to
implement the recommendations. Recommendations often become the
basis for designing educational materials for use in the social marketing of
disease prevention practices. Examples that have been adopted include

•   Condoms and STD health services were introduced in the holding area
    in the Los Angeles County jail for male detainees when the rapid
    assessment procedure identified the lack of condoms as a source of
    increases in STDs.

•   In Florida, a rapid assessment procedure identified gaps in the access
    among members of the Seminole tribe to health care. To improve
    awareness of sociocultural issues among health care providers, the



Page 19                                        GAO-03-455 Federal Ethnography
                       Appendix I: How Ethnographic Studies Can
                       Inform Agencies’ Actions




                          local health department is developing a liaison with alternative health
                          providers from the tribe.

                       These examples focus on site-specific benefits of the rapid assessment
                       procedure. However, its successful use in certain communities has made it
                       potentially useful as a guide for future work in similar communities.
                       DSTDP’s records date only from 2002 and are thus too few, and the sites
                       and populations are too dissimilar, to allow cross-site generalization.
                       However, DSTDP staff indicated that they are accumulating data from
                       urban sites in the hope of eventually being able to do cross-site analysis.
                       They are also developing an evaluation instrument for the rapid
                       assessment procedure that will take into account the method’s feasibility,
                       acceptability, and value in different communities and that will focus on
                       how CDC staff have improved the delivery of public health services.


Census Ethnographic    As the premier source of information about America’s population, the
Studies Inform the     Bureau of the Census strives to produce data that are accurate, timely,
Enumeration of         relevant, and cost-effective. It provides the population counts used to
                       apportion seats in the House of Representatives, among other things. For
Populations That Are   the census, several challenges to accuracy have long been apparent—in
Difficult to Count     particular, the challenges of constructing a comprehensive list of
                       households to survey and of overcoming impediments to participation
                       among certain racial and ethnic groups that have long been undercounted.
                       Through its Statistical Research Division, the Bureau has shed light on
                       these critical problems of data quality by sponsoring a series of studies by
                       in-house and outside ethnographers that have documented undercounts
                       and other enumeration errors, explored factors contributing to them, and
                       led to recommendations for improvements in enumeration practice.

                       The series began in the late 1960s with an ethnographic study of a single
                       block in a low-income urban neighborhood. The study revealed
                       undercounts of adult men who were observed to be living in households
                       but not reported to Census interviewers. The study’s authors concluded
                       that women who filled out Census forms deliberately concealed the
                       presence of adult males in the household to protect household resources
                       such as public assistance and unreported income. However, since only one
                       site was studied, there was no way of knowing how widespread this
                       practice might be. Subsequent ethnographic studies of low-income urban
                       areas and populations provided further evidence of concealment and
                       suggested that ideological resistance to mainstream society, complex and
                       ambiguous living arrangements, and varying beliefs about who should be




                       Page 20                                        GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




considered to be “residing” in a household might also contribute to
undercounts.

To further explore these emerging hypotheses, the 1988 Dress Rehearsal
Census included ethnographic coverage evaluation studies in five sites
encompassing various populations thought likely to be undercounted.
Ethnographers documented the census day residence for all persons
enumerated and compared their observed count to the “official” count for
the same blocks; then ethnographers returned to the field to examine
discrepancies. These studies revealed three major types of error:

•   undercounts from omitted households (persons in residences that were
    not on the Census’s address list or were erroneously listed as vacant),

•   undercounts from omitted individuals (persons present in a residence
    but not included on the household roster), and

•   overcounts from the same person’s being counted in more than one
    household—apparently as a result of extended families whose
    members (such as a mother or grandmother) circulated between
    several neighboring housing units.

However, the small number of sites and the lack of a standard format for
recording data again limited what could be understood from these results.

To assess these findings more systematically as part of the 1990 census,
Census’s Statistical Research Division sponsored ethnographic alternative
enumerations in 29 rural and urban sites, about 100 households each,
encompassing a variety of undercounted racial and ethnic populations.
The work was conducted by ethnographers outside the Bureau who had
already established relationships with the sites. Each study team
developed a complete list of all housing and people living in the area and
recorded logs of systematic behavioral observations about aspects of the
neighborhood, housing units, households, and people in the area that
might have prevented a complete count. After the Bureau matched the
ethnographers’ counts with the count obtained from Census forms
returned by mail, the ethnographers returned to the field to examine
discrepancies and confirm or correct the counts. In addition to providing
alternative counts that revealed enumeration errors, the ethnographers’
studies identified five key factors that contributed to the disparity of
accuracy in census counts for low-income minority populations:




Page 21                                       GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




•    irregular housing units that were missed in the census,

•    complex or ambiguous households that made census residency rules
     difficult to apply,15

•    residential mobility,

•    limited English proficiency, and

•    distrust of government.

The studies also shed light on neighborhood housing and demographic
conditions in which these barriers to accuracy were likely to arise—
potential hot spots for census coverage problems. As illustrated in figure 2,
they yielded recommendations for addressing each factor.




15
 Such households may include unrelated individuals, mobile members, persons whose
only connection is that they share the rent or other costs, or households that contain two
or more nuclear families.




Page 22                                                  GAO-03-455 Federal Ethnography
                                                           Appendix I: How Ethnographic Studies Can
                                                           Inform Agencies’ Actions




Figure 2: Ethnographic Studies and Recommendations on Enumerating Populations Difficult to Count

Community                                                                           Associated
factors identified                               Associated conditions              enumeration error                    Recommendations


Scarcity of affordable                           Irregular or hidden housing        Undercount from missed               Use locals to verify address
conventional housing                             units                              housing units                        lists to Census

                                                                                                                         Train enumerators to seek
                                                                                                                         out hidden units



Mobile population                                Complex households such as         Overcount by reason of               Change definition of
                                                 families spread across several     duplication                          household, residency rules
                                                 housing units

                                                 Ambiguous households that          Undercount from omissions in
                                                 contain marginal members           reported household rosters



Distrust of government                           Motivation to conceal certain                                           Involve community leaders
                                                 residents                                                               and media in promoting the
                                                                                                                         census




Language and literacy                            Limited motivation or ability to   Undercount from nonreturn            Use community-based or
barriers                                         complete Census forms or open      from listed households               culturally sensitive
                                                 door to enumerator                                                      enumerators

                                                                                                                         Make forms and assistance
                                                                                                                         available in community
                                                                                                                         locations

                                                                                                                         Make native-language forms
                                                                                                                         more readily available
Source: GAO analysis of information from Census reports.



                                                           Social scientists at the Bureau joined data from the alternative
                                                           enumeration to data from Census questionnaires to explore clustering
                                                           among the sites and the relative values of explanatory variables—such as
                                                           race, geography, or the relationship of an individual to the householder—
                                                           on the prediction of census omissions and erroneous enumerations. (The
                                                           Bureau had hoped to be able to use data from ethnographers’
                                                           observational logs, but these data were not sufficiently complete to sustain
                                                           analysis.) Related efforts included a Bureau-sponsored 1,000-household
                                                           survey and semi-structured in-depth interviews on the subject of
                                                           residence. Other related efforts were methodological studies testing an
                                                           expanded roster approach that asked respondents to report persons with
                                                           casual or tenuous attachments to the household as well as core members.



                                                           Page 23                                                 GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




The results of the ethnographic studies fed into planning for the 2000
census as well. The complexity of the planning and the many sources of
input make it difficult to trace the influence of any one source. However,
reports from panels of the National Academy of Sciences give evidence
that the ethnographic studies were important in providing knowledge
about the problems of enumerating urban and rural low-income
populations, immigrants, and internal migrants. Like the ethnographic
studies, the National Academy of Sciences reports recommended using
ethnographic methods in hard-to-enumerate areas, such as those
characterized by a shortage of affordable housing, a high proportion of
undocumented immigrants, and low-income neighborhoods. The 2000
census incorporated several procedures that were consistent with findings
and recommendations from the ethnographic studies. The Bureau

•    asked local and tribal governments to review the accuracy and
     completeness of its addresses and field checked the resulting changes;16

•    reworded the instructions for filling out the household roster, which
     now asks how many people were “living or staying” in the house on
     census day;

•    made questionnaires and questionnaire guides available in additional
     languages and at community locations where residents could obtain
     help in filling them out;

•    expanded its partnerships with local organizations, enlisting their
     assistance in recruiting temporary Census workers, providing space
     and volunteers for Questionnaire Assistance Centers, organizing
     promotional events, and customizing informational materials to
     address local residents’ concerns, and generally encouraging people—
     especially hard-to-count populations—to participate in the census;

•    instituted a paid advertising campaign that included local and national
     census advertising; and

•    called on local Census offices to develop plans for following up with
     nonresponding households in hard-to-enumerate areas in their




16
 The Local Update of Census Addresses program was implemented pursuant to the Census
Address List Improvement Act, Pub. L. 103-430.




Page 24                                             GAO-03-455 Federal Ethnography
                             Appendix I: How Ethnographic Studies Can
                             Inform Agencies’ Actions




                                 jurisdictions—for example, using bilingual enumerators in immigrant
                                 neighborhoods or paired/team enumerators in high-crime areas.

                             As we have noted in earlier reports, these practices reflect the Bureau’s
                             recognition that a successful head count requires calling on local
                             knowledge, expertise, and experience. The change to a more localized
                             approach signals a change in the Bureau’s culture, in which centralized
                             control and standardized methods were traditionally paramount. Census’s
                             Statistical Research Division staff saw this change in culture as an
                             important result of the ethnographic studies.

DOD Combined Structured      An example from DOD illustrates the use of ethnographic surveys to
and Ethnographic Surveys     augment information gathered from a national survey and identify factors
of Youths to Better Target   that affect young people’s propensity to join the military. Recruiting and
                             retaining a high-quality, diverse work force is essential to meeting DOD’s
Its Recruiting Program       personnel and readiness goals, and youths are the major source of new
                             entrants into the military. Thus, information about youths’ opinions,
                             attitudes, and beliefs about the military and their propensity to enlist is of
                             vital interest to recruitment commands and DOD personnel managers.
                             Youths’ attitudes toward the military were monitored from 1975 through
                             1999 through a national telephone survey called the Youth Attitudes
                             Tracking Study (YATS). In the mid-1990s, YATS data showed a steep
                             decline in a propensity to join the military across the spectrum of
                             American youths. However, the design of the survey—which measures
                             propensity through exactly worded questions to allow valid comparisons
                             across years and ethnic groups—did not allow probes into why a person
                             responded in a particular manner.

                             To obtain insight into propensity, DOD engaged a social science research
                             firm to conduct a series of ethnographic surveys linked to YATS. The first
                             survey, which focused on young men, illustrates the approach. The study
                             sample was drawn from young men in the prime military recruiting
                             market—high school seniors aged 17 to 21 and high school graduates—
                             who had responded to YATS in fall 1995. It was evenly divided between
                             blacks, whites, and Hispanics and by four categories of propensity, as
                             shown by responses to items in YATS:

                             •   joiners—most likely to join the military;

                             •   nonjoiners—least likely to join the military;

                             •   shifters—had seriously considered joining but changed their minds;




                             Page 25                                          GAO-03-455 Federal Ethnography
Appendix I: How Ethnographic Studies Can
Inform Agencies’ Actions




•   fence-sitters—had given both positive and negative responses about
    joining the military.

The final sample of 120 men was monitored for balance in characteristics
such as age, employment, and geography. Although not statistically
representative, it covered all major areas, groups, and levels of interest in
the military.

Respondents participated in in-depth telephone interviews with a small
number of senior and midlevel researchers. Telephone interviewing was
selected because travel costs would have made face-to-face interviewing
prohibitively expensive and because initial interviews had demonstrated
that rapport could be adequately maintained over the telephone.
Interviewers—who were matched to participants by race and ethnicity—
used a 45-minute structured interview protocol that covered the career
decision-making process, consideration of military enlistment, and
knowledge of the military way of life. The instrument included probes to
clarify or uncover the deeper meaning of responses but was designed to
allow respondents to describe and reflect on their decision processes in
their own words. Interviews were voluntarily taped, and each was
transcribed verbatim. Using iterative procedures similar to those depicted
in figure 1, the researchers reviewed the transcripts, using a set of broad
analytic questions, and reviewed them a second time, looking particularly
at factors associated with propensity. (One analytic question asked
whether there were systematic differences between respondents’ stories
as they considered entering the military as enlisted personnel versus
entering as officers.) Responses to questions about knowledge of the
military were tabulated, as were responses to questions on income that
required a numeric answer.

Similar procedures were used with subsequent studies of young women
(based on 1997 YATS), young Hispanic men (1997 YATS), and parents of
young men who participated in the 1998 YATS. In each case, questions
were adapted to the population being studied. For example, the survey of
women asked about how, if at all, plans for having a family influenced the
respondent’s choice of future activities and about perceptions of
difference in the military experience for young men and young women.
The survey for Hispanics asked about the family’s origin, how many
generations of the family had been in the United States, and whether the
family’s experience with the military in the country of origin had affected
the respondent’s views about military service in the United States. The
parents’ study was prompted by evidence from other DOD reports
concerning the importance of parents as influences on youths’ career


Page 26                                         GAO-03-455 Federal Ethnography
               Appendix I: How Ethnographic Studies Can
               Inform Agencies’ Actions




               decision making. It probed their role in their sons’ career decision making
               and the image of the military that they had and that they conveyed to their
               sons.

               To inform its recruitment advertising, DOD also supported interview
               studies of parents and others, such as educators and family friends, who
               influence youths’ career decisions. These studies, carried out by a different
               firm and reported in 2001 and 2002, were conducted through face-to-face
               interviews and had a qualitative component. Like the ethnographic surveys
               of youths, the surveys of influencers included open-ended questions and
               follow-up probes. However, other aspects of the youth-influencer surveys
               did not reflect ethnography. Their approach was based on theories of
               individual decision making drawn from psychology, and interviewers were
               trained in psychological interviewing. Although noting that social norms
               also drive future behavior, the researchers focused on self-esteem and the
               role of deeply held values, such as personal security and love of family—in
               this case, values that drive influencers’ choices when recommending
               military service or other options to youths.

               Data collection and analysis followed the Values in Strategy Assessment
               (VISTA™) method for mapping the factors important in consumer
               decision making, and the interview guide included many fixed-response
               questions in addition to those that were open-ended. The method was
               designed to support the development of psychological profiles of potential
               users of a product and was used to develop strategies for communication
               campaigns—in this case, DOD recruiting ads aimed at adult influencers.

               The first of these examples from DOD illustrates that using ethnographic
               techniques can follow up on and yield a deeper understanding of data from
               structured surveys. The decision to use telephone rather than face-to-face
               interviewing illustrates how methods can be adapted to fit the
               circumstances. The second example supports our earlier observation that
               qualitative techniques like those used in ethnography are also used in
               studies rooted in other disciplines.


               In reviewing information from our overview and the four case studies, we
Concluding     observed that in each example, ethnography helped obtain previously
Observations   unavailable information about beliefs and behaviors that was important to
               the federal program’s ability to attain its objectives—information that
               could not be readily obtained by other methods. The studies were not
               done merely to add to general knowledge about the groups studied. They
               also addressed program-related issues, such as whether families in Early


               Page 27                                        GAO-03-455 Federal Ethnography
                                              Appendix I: How Ethnographic Studies Can
                                              Inform Agencies’ Actions




                                              Head Start are comfortable with a Montessori approach or how to allocate
                                              fishery harvests fairly among the people affected by them.

                                              Grouping our examples, we identified four major study designs and the
                                              uses they were put to, as shown in table 2. Reviewing this table and the
                                              studies it lists, we offer some concluding observations that may help
                                              agencies and the Congress determine when ethnographic approaches may
                                              help answer questions about a program.

Table 2: Ethnographic Study Designs and Uses

 Nature of study                                                      Example
 Site-specific field studies that address a local issue at a          • CDC studies of STD outbreaks
 particular point in time and support local action to respond to it • DOD study of Antarctic over-wintering
                                                                      • Anthropologists’ work with communities to protect source water for
                                                                        EPA
                                                                      • NMFS community-specific SIAs
                                                                      • NPS studies of culturally meaningful sites
 Site-specific field studies that alert agency staff to a potentially • Early Census studies of low-income urban sites
 widespread phenomenon (or demonstrate the absence of a               • HHS studies of two Montessori sites in EHS evaluation
 hypothesized phenomenon) and to the need for further                 • Justice studies of street gangs
 investigation or action
 Multi-site field studies or observations using a common              • 1990 Census alternative enumeration studies
 framework, intended to support comparisons across sites and • CDC rapid assessment procedure data as information from more
 over time or for program evaluation or planning purposes at            sites becomes available
 the national level. May also support site-specific actions           • Labor evaluation of Youth Opportunity Act initiative
                                                                      • NOAA framework for future community profiles
                                                                      • Observations on drug use fed into Pulse Check for ONDCP
 Interview studies to explore more deeply beliefs or behaviors        • Census cognitive interviewing and ethnographic survey regarding
 suggested by previous research and (if based on a systematic           residency terms
 sample) to produce results that can be generalized to a broad • DOD ethnographic surveys with YATS respondents
 population and inform program changes at the national level
Source: GAO analysis.

                                              Single-site ethnographic field studies can be valuable for addressing
                                              problems specific to a site and for raising red flags—discovering
                                              phenomena important to a program that may be widespread and that may
                                              merit further investigation. Combined with other evidence from a larger
                                              sample, field studies of small numbers of sites can also help disconfirm a
                                              suspected problem (such as discomfort with the Montessori approach to
                                              early childhood education). However, such studies have an important
                                              limitation. Findings from a single site or a small number of sites cannot be
                                              generalized to a broader population and, thus, are not sufficient to inform
                                              change in a national program.

                                              The limitations of single-site field studies can be overcome by
                                              (1) obtaining sufficient site-specific studies to reliably identify recurring
                                              themes or issues and (2) arranging for field researchers to gather and



                                              Page 28                                                  GAO-03-455 Federal Ethnography
           Appendix I: How Ethnographic Studies Can
           Inform Agencies’ Actions




           report a common core of data in terms of a common framework, while
           leaving them open to the discovery of additional information. Having this
           common core facilitates cross-site analysis, site analysis over time, and
           linking ethnographic to economic or other data.

           •   Information from multiple field sites can be obtained by design or
               accretion. For the 1990 alternative enumeration studies, the Bureau of
               the Census selected the 29 sites to ensure coverage of the range of
               groups and living situations that were difficult to enumerate. CDC and
               NMFS sought to achieve broad coverage over time as community
               studies accumulated.

           •   One way to build in a common framework is to have in-house staff do
               the fieldwork, as at CDC. For agencies that rely on outside experts
               accustomed to working independently, commonality may be more
               difficult to achieve. Some of the ethnographers conducting the 1990
               alternative census enumerations did not collect the full set of data the
               Bureau had asked for, and the aggregate data set was not sufficiently
               complete to sustain analysis. Whether NMFS will succeed in obtaining
               specified core data in profiles of fishing communities remains to be
               seen. However, NMFS has more than a decade of experience and
               studies to draw on for crafting guidelines, as well as long experience
               with the academic experts and contract firms that conduct SIAs. This
               experience, along with increased NMFS staffing and program direction,
               may make a difference.

           When initial information about a situation is available from previous
           fieldwork or other research, and the population of interest is scattered
           geographically, an ethnographic survey of a systematic sample of that
           population can be used to obtain broadly applicable findings that can
           inform decisions about national programs.




(460538)
           Page 29                                        GAO-03-455 Federal Ethnography
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548