oversight

Child Welfare: Most States Are Developing Statewide Information Systems, but the Reliability of Child Welfare Data Could Be Improved

Published by the Government Accountability Office on 2003-07-31.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

             United States General Accounting Office

GAO          Report to Congressional Requesters




July 2003
             CHILD WELFARE

             Most States Are
             Developing Statewide
             Information Systems,
             but the Reliability of
             Child Welfare Data
             Could Be Improved




GAO-03-809
                                                July 2003


                                                CHILD WELFARE

                                                Most States Are Developing Statewide
Highlights of GAO-03-809, a report to           Information Systems, but the Reliability
Congressional Requesters
                                                of Child Welfare Data Could Be Improved



To better monitor children and                  HHS reported that 47 states are developing or operating a SACWIS, but many
families served by state child                  continue to face challenges developing their systems. Most state officials
welfare agencies, Congress                      said they recognize the benefit their state will achieve by developing
authorized matching funds for the               SACWIS, such as contributing to the timeliness of child abuse and neglect
development of statewide                        investigations; however, despite the availability of federal funds since 1994,
automated child welfare
information systems (SACWIS) and
                                                states reported a median delay of 2-½ years beyond the timeframes they set
required that the Department of                 for completion. States reported that they encountered some difficulties
Health and Human Services (HHS)                 during SACWIS development, such as challenges receiving state funding and
compile information on the                      creating a system that reflected their work processes. In response to some of
children served by state agencies.              these challenges, HHS has provided technical assistance to help states
This report reviews (1) states’                 develop their systems and conducted on-site reviews of SACWIS to verify
experiences in developing child                 that the systems meet federal requirements.
welfare information systems and
HHS’s role in assisting in their                Despite efforts to implement comprehensive information systems, several
development, (2) factors that affect            factors affect the states’ ability to collect and report reliable adoption, foster
the reliability of data that states             care, and child abuse and neglect data. States responding to GAO’s survey
collect and report on children
served by their child welfare
                                                and officials in the 5 states GAO visited reported that insufficient
agencies and HHS’s role in                      caseworker training and inaccurate and incomplete data entry affect the
ensuring the reliability of those               quality of the data reported to HHS. In addition, states reported technical
data, and (3) practices that child              challenges reporting data. Despite HHS’s assistance, many states report
welfare agencies use to overcome                ongoing challenges, such as the lack of clear and documented guidance on
challenges associated with SACWIS               how to report child welfare data. In addition, although states were mandated
development and data reliability.               to begin reporting data to the Adoption and Foster Care Analysis and
                                                Reporting System (AFCARS) in 1995, few reviews of states’ AFCARS
                                                reporting capabilities have been conducted to assist states in resolving some
In order to improve the reliability             of their reporting challenges.
of state-reported child welfare
data, GAO recommends that the                   Some states are using a variety of practices to address the challenges
Secretary of HHS consider ways to               associated with developing SACWIS and improving data reliability. For
enhance the guidance and                        example, 44 states included caseworkers and other system users in the
assistance offered to states to help            design and testing of SACWIS, and 28 states reported using approaches to
them overcome the key challenges                help caseworkers identify and better understand the data elements that are
in collecting and reporting child               required for federal reporting.
welfare data. In commenting on
this report, HHS generally agreed
                                                Factors That Affect the Reliability of Data Reported to HHS
with GAO’s findings and
commented that the report                         Factors affecting data quality:                                Which may lead to
provides a useful perspective of the                                                                             inaccurate measures of:
                                                                                                       i  on
problems states face in collecting                   Inaccurate and incomplete data-                 at
                                                                                                 rm
                                                     entry by caseworkers                  In
                                                                                              fo                  State performance
data and of HHS’s effort to provide                                                                               on federal outcomes
                                                     Insufficient caseworker training
ongoing technical assistance to                      Differences between state and                                Children's
improve child welfare data.                          federal data definitions
                                                                                        Resulting in              experiences,
                                                                                        potentially unreliable    such as time
                                                     Lack of clear, documented
www.gao.gov/cgi-bin/getrpt?GAO-03-809.                                                  information on abused     spent in
                                                     guidance from HHS
                                                                                        and neglected             foster
                                                     Difficulty accessing technical
To view the full product, including the scope                                           children available in     care
                                                     assistance from HHS
and methodology, click on the link above.                                               federal data systems
For more information, contact Cornelia M.
                                                Source: GAO analysis.
Ashby at (202) 512-8403 or
AshbyC@gao.gov.
Contents


Letter                                                                                             1
                       Results in Brief                                                            2
                       Background                                                                  5
                       Most States Are Developing SACWIS, but Challenges Remain
                         Despite HHS’s Oversight and Technical Assistance                        10
                       Several Factors Affect the States’ Ability to Ensure Reliable Data
                         on Children’s Experiences, and Some of HHS’s Oversight and
                         Assistance Is Problematic                                               24
                       States Are Using Various Practices to Overcome System
                         Development Challenges and Improve Data on Children’s
                         Experiences                                                             38
                       Conclusion                                                                42
                       Recommendation to the Secretary of Health and Human Services              43
                       Agency Comments                                                           43

Appendix I             Scope and Methodology                                                      47



Appendix II            State SACWIS Stages of Development                                         49



Appendix III           Comments from the Department of Health and
                       Human Services                                                             50



Appendix IV            GAO Contacts and Acknowledgments                                           62
                       GAO Contacts                                                              62
                       Staff Acknowledgments                                                     62

Bibliography                                                                                      63



Related GAO Products                                                                              64



Tables
                       Table 1: Significant Child Welfare Information and Data Systems             6



                       Page i                                               GAO-03-809 Child Welfare
          Table 2: Number of States in Various Stages of SACWIS
                   Development                                                                      12
          Table 3: Selected SACWIS Child Welfare and Administrative
                   Services                                                                         14
          Table 4: Commonly Used SACWIS Development Participants and
                   Their Level of Helpfulness                                                       15
          Table 5: State-Reported Benefits of SACWIS Development                                    17
          Table 6: Number of Months States Delayed in SACWIS
                   Development                                                                      18
          Table 7: Common SACWIS Development Challenges                                             19
          Table 8: Selected AFCARS Elements and Six States’ Levels of
                   Compliance in Meeting Reporting Requirements                                     32


Figures
          Figure 1: Most Common Caseworker Issues That Affect Data
                   Quality                                                                          26
          Figure 2: Federal Practices That Affect Data Quality                                      35

          Abbreviations

          ACF               Administration for Children and Families
          ADP               advance planning document
          AFCARS            Adoption and Foster Care Analysis and Reporting System
          CAPTA             Child Abuse Prevention and Treatment Act
          CRSR              Child and Family Services Reviews
          FFP               federal financial participation
          HHS               Department of Health and Human Services
          IT                information technology
          NCANDS            National Child Abuse and Neglect Data System
          NYTD              National Youth in Transition Database
          SACWIS            Statewide Automated Child Welfare Information System
          VCIS              Voluntary Cooperative Information System



          This is a work of the U.S. Government and is not subject to copyright protection in the
          United States. It may be reproduced and distributed in its entirety without further
          permission from GAO. It may contain copyrighted graphics, images or other materials.
          Permission from the copyright holder may be necessary should you wish to reproduce
          copyrighted materials separately from GAO’s product.




          Page ii                                                        GAO-03-809 Child Welfare
United States General Accounting Office
Washington, DC 20548




                                   July 31, 2003

                                   The Honorable Charles E. Grassley
                                   Chairman, Committee on Finance
                                   United States Senate

                                   The Honorable Tom DeLay
                                   Majority Leader
                                   House of Representatives

                                   Recent news reports of tragedies involving child welfare agencies across
                                   the country highlight the long-standing problems states have had
                                   monitoring children in their care. Given that hundreds of thousands of
                                   children are found to be victims of abuse and neglect and are estimated to
                                   spend some time in foster care each year, the Congress required that the
                                   Department of Health and Human Services (HHS) compile information on
                                   the children served by state agencies and authorized federal funds to
                                   match those of states for use in the development of state child welfare
                                   information systems. Since 1994, designated federal matching funds have
                                   been available to states to develop and implement comprehensive case
                                   management systems—Statewide Automated Child Welfare Information
                                   Systems (SACWIS)—to manage their child welfare cases as well as to
                                   report child abuse and neglect, foster care, and adoption information to
                                   the federal government. States have the option to implement a SACWIS or
                                   develop different information systems without using SACWIS funds to
                                   support their child welfare agencies and collect information on their child
                                   welfare cases. Regardless of the type of system a state develops, child
                                   welfare caseworkers at the county or local level are the key personnel
                                   who collect and document information on children and families served by
                                   child welfare agencies, in addition to performing a wide range of services
                                   to protect children—such as investigating child abuse or neglect reports or
                                   providing support services to maintain the children in their homes.
                                   Currently, HHS compiles state-reported child welfare data in two
                                   databases: the Adoption and Foster Care Analysis and Reporting System
                                   (AFCARS) and the National Child Abuse and Neglect Data System
                                   (NCANDS). HHS relies on the information available in its databases to
                                   analyze and track children’s experiences in the child welfare system, to
                                   determine states’ performance on federal child welfare outcome measures,
                                   and to report to Congress on children’s well being and child welfare
                                   experiences.



                                   Page 1                                              GAO-03-809 Child Welfare
                   Because of your interest in how states have developed systems to collect
                   and report information on the children they serve and the reliability of that
                   information, you asked us to determine (1) states’ experiences in
                   developing child welfare information systems and HHS’s role in assisting
                   in their development; (2) factors that affect the reliability of data that
                   states collect and report on children served by their child welfare
                   agencies, and HHS’s role in ensuring the reliability of those data; and
                   (3) practices that child welfare agencies use to overcome challenges
                   associated with SACWIS development and data reliability.

                   To conduct our work, we surveyed all 50 states and the District of
                   Columbia regarding their experiences in developing and using information
                   systems and their ability to report data to HHS. We received responses
                   from 49 states and the District of Columbia,1 although all states did not
                   respond to every question. We also reviewed a variety of HHS documents,
                   including the protocol and reports for its reviews of SACWIS systems and
                   states’ AFCARS reporting capabilities. In addition, we visited 5 states—
                   Colorado, Iowa, New York, North Carolina, and Oklahoma—to obtain first-
                   hand information on their experiences developing SACWIS and reporting
                   data to HHS. We selected these states to represent geographic diversity
                   and different stages of SACWIS implementation. Finally, we interviewed
                   HHS officials and child welfare and data experts and reviewed relevant
                   literature. We conducted our work between June 2002 and June 2003 in
                   accordance with generally accepted government auditing standards. A
                   more detailed discussion of our scope and methodology appears in
                   appendix I.


                   HHS reported that 47 states are developing or operating a SACWIS, but
Results in Brief   many continue to face challenges despite HHS’s oversight and technical
                   assistance. Most states are using federal SACWIS funds and are in various
                   stages of development. States reported in our survey that they have spent
                   approximately $2.4 billion in federal, state, and local funding on SACWIS.
                   Most state officials said they recognize the benefit their state will achieve
                   by developing SACWIS, such as contributing to the timeliness of child
                   abuse and neglect investigations; however, despite the availability of
                   federal funds since 1994, many child welfare agencies lag behind the


                   1
                    Throughout this report, references to state survey responses include the District of
                   Columbia. Forty-six of these states reported that they are developing or operating a
                   SACWIS. Nevada, which HHS reported has an operational SACWIS, did not respond to our
                   survey.




                   Page 2                                                      GAO-03-809 Child Welfare
timeframes they set for completion, with delays ranging from 2 months to
8 years. Forty-two of the 46 states responding to our survey that they are
developing SACWIS reported at least some challenge obtaining state
funding. In Iowa, for example, state officials reported that insufficient
state funds delayed them in making the necessary modifications to meet
federal requirements for system completion. Some states had difficulties
developing a system that met the state child welfare agency’s needs
statewide. For example, state officials in New York—a state where the
counties are responsible for administering child welfare services—said
that building a uniform system was stalled when significant frustration
with the system’s design led some county officials to request that the state
stop SACWIS development. In addition, 32 states reported at least some
challenge securing information technology contractors with knowledge of
child welfare practice to develop their SACWIS. In response to some of
these challenges, HHS has provided technical assistance to help states
develop their systems and conducted on-site reviews of SACWIS to verify
that the systems meet all federal requirements. For example, at the time of
our review, HHS had conducted on-site reviews in 26 states with
operational SACWIS to ensure that the systems met all federal
requirements and to offer assistance to states that faced challenges
completing the development of their SACWIS.

Several factors affect states’ ability to collect and report reliable data on
children served by state child welfare agencies, and some problems exist,
such as a lack of clear and documented guidance, with HHS’s oversight
and technical assistance. Almost all of the states responding to our survey
reported that insufficient caseworker training and inaccurate and
incomplete data entry into their information system affect the quality of
the data reported to HHS. Although most states reported these as separate
factors, HHS and the states we visited found that insufficient training and
inaccurate and incomplete data entry are often linked. In addition, 36 of
the 50 states that responded to our survey reported that technical
challenges, such as matching their state data element definitions to HHS’s
data categories, affected the quality of the data that they report to the
federal government. Similarly, during assessments of 6 states’ compliance
with AFCARS reporting standards, HHS found that these issues affect data
reliability. Despite HHS’s assistance in helping states improve their data,
such as testing state data quality and providing the results to the states to
aid them in resubmitting data, states report ongoing challenges. For
example, 41 of the 50 states responding to our survey reported that a lack
of clear and documented guidance from HHS affects the quality of the data
they report to AFCARS, and 25 states said the lack of clear, documented
guidance also affected the data reported to NCANDS. In addition, although


Page 3                                               GAO-03-809 Child Welfare
states were mandated to begin reporting AFCARS in 1995, few reviews of
states AFCARS reporting capabilities have been conducted to assist states
in resolving some of their challenges.

Some states are using a variety of practices to address the challenges
associated with developing SACWIS and improving data reliability,
although no formal evaluations are available on their effectiveness. To
overcome the challenge of developing a system to meet statewide needs,
many states relied on caseworkers and other system users for input on
design and testing of SACWIS. Few states reported that they devised
strategies to overcome the other challenges, such as limited funding and
finding contractors with knowledge of child welfare. However, Oklahoma
child welfare officials—in order to maximize the limited state funding for
maintaining their SACWIS—reported saving $1 million each year by hiring
some of the contractors who developed their SACWIS as permanent staff.
To improve data reliability, the 5 states we visited routinely review their
data to identify data entry errors so that managers can ensure that the
missing data are entered appropriately. In addition, some states reported
that frequent use of the data, such as publishing periodic management
reports detailing local offices’ performance on outcome measures, helps
caseworkers understand the importance of entering timely information.

In order to improve the reliability of state-reported child welfare data, we
are recommending that the Secretary of HHS consider ways to enhance
the guidance and assistance offered to states to help them overcome the
key challenges in collecting and reporting child welfare data. In
commenting on this report, HHS’s Administration for Children and
Families (ACF) generally agreed with our findings and commented that
the report provides a useful perspective of the problems states face in
collecting data and of ACF’s effort to provide ongoing technical assistance
to improve the quality of child welfare data. In response to our
recommendation, ACF said that we did not recognize the long-term efforts
it has taken to provide AFCARS and NCANDS guidance. ACF also noted
that the data definitions need to be updated and revised and said it is
currently in the process of revising the AFCARS regulations to further
standardize the information states are to report—which we acknowledge
in our report. Further, ACF added that although staff turnover in state
child welfare agencies is a significant contributor to data quality issues, we
did not focus on this as a significant factor. However, because we recently
issued a detailed report on a variety of caseworker issues, we primarily
focused in this report on the key data entry challenges caseworkers face
and refer readers to our previous work for additional information on
challenges related to caseworker recruitment and retention and their


Page 4                                                GAO-03-809 Child Welfare
             affect on child welfare agencies. ACF commented that it is firmly
             committed to continue to support the states and to provide technical
             assistance and other guidance as its resources will permit. We believe that
             the recent activities to formally obtain, document, and incorporate
             feedback from the states with regard to collection and reporting adoption
             and foster care data represent are a step towards improving states’ data.
             Our recommendation encourages HHS to consider ways to enhance the
             guidance and assistance already offered to states as a step to helping them
             better comply with the reporting requirements.


             ACF’s Children’s Bureau is responsible for the administration and
Background   oversight of federal funding to states for child welfare services under
             Titles IV-B and IV-E of the Social Security Act. However, the monitoring of
             children served by state child welfare agencies is the responsibility of the
             state agencies that provide the services to these children and their
             families. Child welfare caseworkers at the county or local level are the key
             personnel responsible for documenting the wide range of services offered
             to children and families, such as investigations of abuse and neglect;
             treatment services offered to families to keep them intact and prevent the
             need for foster care; and arrangements made for permanent or adoptive
             placements when children must be removed from their homes.
             Caseworkers are supported by supervisors who typically assign new cases
             to workers and monitor caseworkers’ progress in achieving desired
             outcomes, analyzing and addressing problems, and making decisions
             about cases.

             A number of efforts at the national level have been taken to implement
             comprehensive data systems that capture, report, and analyze the child
             welfare information collected by the states (see table 1 for information on
             national data systems as well as information on state systems).




             Page 5                                               GAO-03-809 Child Welfare
Table 1: Significant Child Welfare Information and Data Systems

System                                            History                                          Use
State information systems
Statewide Automated Child Welfare Information     The Omnibus Budget Reconciliation Act            A SACWIS is designed and
System (SACWIS)                                   (OBRA) of 1993 authorized the use of an          developed for use by states’
                                                  enhanced federal financial participation         caseworkers and other
                                                  (FFP) rate of 75 percent to assist states        personnel for the purposes of
                                                  develop uniform automated information            establishing an electronic case
                                                  systems—SACWIS—that support the                  file for children and families
                                                  administration of services offered under their   served by the state child welfare
                                                  child welfare programs.a The enhanced FFP        agency. Some of the data
                                                  was available initially from federal fiscal      captured in SACWIS are
                                                  years 1994 through 1996 and subsequently         reported to HHS.
                                                  extended through federal fiscal year 1997.
                                                  After 1997, states receive a 50 percent
                                                  match for SACWIS-related activities.
                                                  Funding approval for SACWIS is based on
                                                  states’ estimated costs for development and
                                                  operation, and no time limits are placed on
                                                  the receipt of federal funding.
National data systems
Voluntary Cooperative Information System (VCIS)   With funding from HHS, the American Public       The data available in VCIS was
                                                  Welfare Association—now known as the             used to produce reports at the
                                                  American Public Human Services                   national level on the
                                                  Association—established VCIS in an effort        characteristics of children in
                                                  to compile state-specific child welfare data.    foster care and those adopted
                                                  States voluntarily reported aggregate-level      from state child welfare
                                                  data on the characteristics of children in       agencies. However, the data
                                                  foster care and those adopted from state         reported to VCIS were
                                                  child welfare agencies. VCIS compiled data       inconsistent.b Following the
                                                  between 1982 and 1994.                           implementation of AFCARS,
                                                                                                   states no longer reported to
                                                                                                   VCIS.




                                         Page 6                                                          GAO-03-809 Child Welfare
System                                            History                                         Use
Adoption and Foster Care Analysis and Reporting   In 1986, Congress added Section 479 to          Some of the key uses of
System (AFCARS)                                   Title IV-E of the Social Security Act, which    AFCARS data include:
                                                  required HHS to establish and implement a
                                                  mandatory foster care and adoption data
                                                  collection system. Despite the legislative      Calculating the national standard
                                                  requirement that HHS complete final             for five federal outcome
                                                  regulations for the system by the end of        measures—incidence of child
                                                  calendar year 1988, regulations were not        abuse and/or neglect in foster
                                                  issued until December 1993. States began        care, foster care re-entries,
                                                  reporting to AFCARS in 1995 on the              stability of foster care
                                                  demographic characteristics of adoptive and     placements, length of time to
                                                  foster children and their parents as well as    achieve reunification, and length
                                                  foster children’s type of placement and         of time to achieve adoption.
                                                  permanency goals. States are required to
                                                  report 66 foster care and 37 adoption data
                                                  elements for each child in care during          The publication of the annual
                                                  6-month periods. Between 1998 and               Child Welfare Outcomes Report.
                                                  2001, states not meeting certain reporting
                                                  standards were subject to penalties. The        Determining the allocation of
                                                  penalties were rescinded in January 2002        funds in the adoption incentive
                                                  following an appeal filed by 12 states.         program and Chafee Foster Care
                                                                                                  Independence Program.c
National Child Abuse and Neglect Data System      In 1988, Congress amended the Child             Some of the key uses of
(NCANDS)                                          Abuse Prevention and Treatment Act              NCANDS data include:
                                                  (CAPTA) by directing HHS to establish a
                                                  national data collection and analysis system
                                                  consisting of state child abuse and neglect     Calculating the national standard
                                                  information. Walter R. McDonald &               for two federal outcome
                                                  Associates was awarded the contract to          measures on child safety—
                                                  compile and analyze the state-reported data.    recurrence of maltreatment and
                                                  States began voluntarily reporting annually     incidence of child abuse and/or
                                                  in 1990; however, the 1996 CAPTA                neglect in foster care.
                                                  amendments directed states to report
                                                  certain data to receive CAPTA grant
                                                  funding. States submit either child-specific    The publication of the annual
                                                  records or aggregate-level data. In 2001—       Child Maltreatment Report.
                                                  the most recently available data from HHS—
                                                  39 states and the District of Columbia          The publication of the annual
                                                  submitted child-specific data, which includes   Child Welfare Outcomes Report.
                                                  the demographics of the children and their
                                                  perpetrators, the types of maltreatment,
                                                  investigation or assessment findings, risk
                                                  factors, and services provided as a result of
                                                  the investigation or assessment. The
                                                  remaining 11 states submitted aggregate
                                                  data.




                                         Page 7                                                         GAO-03-809 Child Welfare
 System                                                                          History                                         Use
 National Youth in Transition Database (NYTD)                                    In response to requirements included in the     In response to the legislation,
                                                                                 Foster Care Independence Act of 1999,           data available in the NYTD will
                                                                                 HHS has proposed to develop the NYTD to         be used, in part, to track (1) the
                                                                                 capture more detailed data than is currently    number and characteristics of
                                                                                 reported to AFCARS on older foster children     older youth receiving
                                                                                 who receive independent living services.        independent living services,
                                                                                 Such services include life-skills training in   (2) the type and quantity of
                                                                                 financial management or career planning.        services states provided, and
                                                                                 HHS reported to Congress in September           (3) state performance on the
                                                                                 2001 that nationwide implementation of          outcome measures required by
                                                                                 NYTD was planned for October 2003, but          the legislation.
                                                                                 recent HHS information indicates that the
                                                                                 actual implementation date will likely be
                                                                                 2005 or 2006. Once implemented, states will
                                                                                 be required to report to NYTD.
Source: GAO analysis based on federal legislation, regulations, and other documents.
                                                                  a
                                                                   The allowable costs under the 75 percent FFP included planning, design, development, and
                                                                  installation of a SACWIS. Other costs, such as operation of the SACWIS, were covered at 50 percent.
                                                                  b
                                                                   See www.acf.hhs.gov/programs/cb/dis/vcis/maintoc.htm for more complete details on the
                                                                  shortcomings of the data available in VCIS.
                                                                  c
                                                                   The total number of finalized adoptions reported to AFCARS are used to determine the amount, if
                                                                  any, of the adoption incentive payments awarded to states. In addition, AFCARS data are used to
                                                                  determine the ratio of the number of children in foster care in each state to the total number of
                                                                  children in foster care in all states to calculate state allotments under the Chafee Foster Care
                                                                  Independence Program.


                                                                  To qualify for federal funding for SACWIS, states must prepare and submit
                                                                  an advance planning document (APD) to ACF’s Children’s Bureau, in
                                                                  which they describe the state’s plan for managing the design, development,
                                                                  implementation, and operation of a SACWIS that meets federal
                                                                  requirements and state needs in an efficient, comprehensive, and cost-
                                                                  effective manner.2 In addition, the state must establish SACWIS and
                                                                  program performance goals in terms of projected costs and benefits in the
                                                                  APD. States are required to submit separate APDs for the planning and
                                                                  development phases, in addition to periodic updates.




                                                                  2
                                                                   When states choose to develop information systems that include other human services,
                                                                  such as food stamps, child support enforcement, or Medicaid, states must submit APDs to
                                                                  each cognizant federal agency. In a hearing held last year before the Subcommittee on
                                                                  Technology and Procurement Policy, House Committee on Government Reform, we
                                                                  testified that the federal agencies do not have systems to monitor states requests for
                                                                  federal approval and funding through the life cycle of a state request. (See U.S. General
                                                                  Accounting Office, Human Services: Federal Approval and Funding Processes for States’
                                                                  Information System, GAO-02-347T (Washington, D.C: July 9, 2002.))




                                                                  Page 8                                                                GAO-03-809 Child Welfare
    Since the administration and structure of state child welfare agencies vary
    across the nation, states can design their SACWIS to meet their state
    needs, as long as states meet certain federal requirements. Federal funding
    is available to states for SACWIS that

•   meet the requirements for reporting AFCARS data to HHS;
•   to the extent practicable, are capable of linking with the state data
    collection system that collects information on child abuse and neglect;
•   to the extent practicable, are capable of linking with, and retrieving
    information from, the state data collection system that collects
    information on the eligibility of individuals under Title IV-A—Temporary
    Assistance for Needy Families; and
•   provides for more efficient, economical, and effective administration of
    the programs carried out under a state’s plans approved under Titles IV-B
    and IV-E of the Social Security Act.

    A SACWIS must operate uniformly as a single system in each state and
    must encompass all entities that administer programs provided under
    Titles IV-B and IV-E. In some cases, HHS will allow the statewide system
    to link to another state system to perform required functions, such as
    linking to financial systems to issue and reconcile payments to child
    welfare service providers. The state’s APD must describe how its SACWIS
    will link to other systems to meet the requirements in the SACWIS
    regulations.

    In addition to monitoring the APDs of the states that are developing
    SACWIS, HHS reviews state information systems through formal SACWIS
    assessment reviews and the Child and Family Services Reviews (CFSR)—a
    federal review process to monitor states’ compliance with child welfare
    laws and federal outcome measures. The formal SACWIS reviews are
    conducted by ACF’s Children’s Bureau to determine if a state has
    developed and implemented all components detailed in the state’s APD
    and if the system adheres to federal requirements. The CFSR assesses
    statewide information systems, along with other systemic factors, to
    determine if the state is operating a system that can readily identify the
    status, demographic characteristics, location, and goals for placement of
    every child who is in foster care. This systemic factor is reviewed in all
    states, regardless of whether the state is developing a SACWIS or the stage
    of system development. According to results from the fiscal years 2001 and
    2002 CFSRs, 4 of the 32 states in which HHS reviewed were not in
    substantial conformity on the statewide information system indicator.
    These 4 states must address how they will come into conformity with this




    Page 9                                              GAO-03-809 Child Welfare
                           factor in a program improvement plan. HHS has also conducted SACWIS
                           reviews in 2 of these states.


                           While 47 states are developing or operating a SACWIS, many challenges
Most States Are            remain despite HHS’s oversight and technical assistance. Since 1994,
Developing SACWIS,         states reported that they have spent approximately $2.4 billion in federal,
                           state, and local funding on SACWIS. While most state officials we
but Challenges             interviewed and those responding to our survey said that they recognize
Remain Despite HHS’s       the benefits their state will achieve by developing a statewide system,
                           many states reported that the development of their SACWIS is delayed
Oversight and              between 2 months and 8 years beyond the time frames the states set for
Technical Assistance       completion, with a median delay of 2-½ years. Most states responding to
                           our survey faced challenges, such as obtaining state funding and
                           developing a system that met the child welfare agency’s needs statewide.
                           In response to some of these challenges, HHS has provided technical
                           assistance to help states develop their systems and conducted on-site
                           SACWIS reviews to verify that the systems meet all federal requirements.


States Are Using Federal   Currently, 47 states are developing or operating a SACWIS and are in
and State Funds and        various stages of development—ranging from planning to complete. The
Various Participants to    states responding to our survey reported using approximately $1.3 billion
                           in federal funds3 and approximately $1.1 billion in state and local funds4 for
Develop MultiComponent     their SACWIS. However, HHS estimated that it allocated approximately
SACWIS                     $821 million between fiscal years 1994 and 2001 in SACWIS developmental
                           funds5 and $173 million between fiscal years 1999 and 2001 in SACWIS




                           3
                            Forty-four states provided information on the total amount of federal funds they received
                           to develop and operate SACWIS. Alaska, Hawaii, Missouri, North Carolina, Texas, and
                           Vermont did not report federal funding information. Nevada did not respond to our survey.
                           State-reported figures may include some funding allocated in fiscal year 2003 since the
                           survey was issued in October 2002 and completed as late as December 2002.
                           4
                            Forty-four states provided information on the total amount of state funds used to develop
                           and operate SACWIS. Arkansas, Hawaii, Missouri, North Carolina, Texas, and Vermont did
                           not report state funding information. Nevada did not respond to our survey. State-reported
                           figures may include some funding allocated in fiscal year 2003 since the survey was issued
                           in October 2002 and completed as late as December 2002.
                           5
                           This figure includes developmental funds allocated by HHS to 49 states and the District of
                           Columbia. Hawaii did not take any federal money for SACWIS development.




                           Page 10                                                        GAO-03-809 Child Welfare
operational funds.6 The total amount of federal funding provided to states
for SACWIS is unknown because states claimed operational costs as a part
of their Title IV-E administrative expenses prior to 1999.7 Although the
federal government matched state funding at an enhanced rate of
75 percent beginning in 1994, many states did not apply for federal funding
or begin SACWIS development until 1996 and 1997 when more than
$467 million—the bulk of federal funds—were allocated. Most states were
still developing their SACWIS by the time enhanced funding expired in
1997, after which states could receive a 50 percent FFP for SACWIS
development and operation. Although 47 states are currently developing or
operating a SACWIS, all states except Hawaii received some federal
SACWIS funds. For example, according to figures provided by HHS, North
Carolina received approximately $9.6 million in developmental funds and
North Dakota received approximately $2.4 million in developmental funds
and $245,000 in operational funds for SACWIS, but both states
encountered difficulties that prevented them from completing their
systems. In these situations, HHS entered into negotiations with the states
about the amount of money that the states must return to the federal
government.

In order to track states’ SACWIS development, HHS places them in six
categories that identify their stage of development (see table 2). States are
required to submit APD updates periodically, which inform HHS of their
progress in developing SACWIS. See appendix II for a complete list of
states’ phases of development. Although most states continue to advance
in the development of their systems, some encounter problems that cause
HHS to recategorize them into a lower stage of development. In
Pennsylvania, for example, the child welfare agency encountered
difficulties, such as inadequate computer software to support a
comprehensive SACWIS, after attempting to implement its SACWIS in


6
 This figure includes operational funds allocated to 35 states. States begin claiming
operational costs when some or all components of their SACWIS are operating in local
offices. Operational activities include routine maintenance, minor enhancements, and other
changes that do not significantly increase or modify the functionality of the system.
7
 According to HHS officials, prior to fiscal year 2000, states reported SACWIS operational
expenses as part of their Title IV-E administrative expenses because the claims sheet states
used for reporting did not have a separate column for SACWIS operational expenditures. In
fiscal year 2000, states were required to use a claims sheet that was reformatted to provide
space for SACWIS operational expenditures. In addition, an HHS official explained that the
difference between the state-reported figures and the federal figures may be due to states
claiming some SACWIS expenses under different programs, such as Title IV-E
administrative funds, rather than separately as SACWIS expenses.




Page 11                                                         GAO-03-809 Child Welfare
2000. Due to these problems, the state is in the process of shutting down
the system and has resubmitted an APD for a new system to HHS for
review and approval for further federal funding. According to figures
provided by HHS, Pennsylvania has received approximately $9.7 million in
federal funding thus far. In addition, while HHS may classify a state system
as complete following an assessment of their SACWIS, a state may make
additional changes to the system since SACWIS, like other computer
systems, continually evolve as technology and child welfare practices
change. States can claim federal funding for these changes as operational
expenses. For example, Oklahoma’s SACWIS was the first system to be
determined complete, but it has made enhancements to its system since
HHS found the system in compliance with federal requirements in 1998. In
addition, Oklahoma is currently considering moving to a Web-based
system. An HHS official reported that such changes do not need prior
approval unless they are in excess of $5 million.

Table 2: Number of States in Various Stages of SACWIS Development

    Stage                                                                               Number of states
    Completea                                                                                                 5
                 b
    Operational                                                                                               24
                             c
    Partially operational                                                                                     9
                         d
    Implementation                                                                                            2
    Planninge                                                                                                 7
                     f
    No SACWIS                                                                                                 4
Source: HHS.

Note: Status is as of May 22, 2003.
a
 SACWIS assessment process is completed, and all functional requirements and specifications set
forth in the APD are either included in the system or in an accepted corrective action plan.
b
 All functional requirements and specifications in APD are included in system, and system is
functional statewide, but state has not completed SACWIS assessment or is working on other issues.
c
 State is still rolling out system to field sites or still adding functions to systems that are operational
statewide.
d
    In active design and development, even if delayed while waiting to resolve problems such as funding.
e
    Working through options for a SACWIS.
f
    Have never pursued SACWIS funding or have abandoned plans to develop a system.


In developing a system, states have considerable flexibility in the design of
their SACWIS. According to HHS officials, a state should be using its
SACWIS as a case management tool that uses automation to support the
various aspects of state child welfare programs, such as recording child



Page 12                                                                      GAO-03-809 Child Welfare
protection, out of home care, and foster care and adoption services. To
further assist child welfare practice, states have designed their systems to
follow the natural flow of child welfare practice in their state and have
added design features to help track key events during a case. For example,
in Iowa child welfare work is divided between child abuse and neglect
investigations and ongoing case management for children brought into the
care of the child welfare agency. As a result, Iowa designed a SACWIS to
reflect this work process by linking two databases—one to record child
abuse and neglect information and one to record ongoing case records—
that share information with one another.8 In Rhode Island, the SACWIS
was designed to alert caseworkers if an alleged perpetrator has been the
subject of three reports of abuse or neglect within 1 year. Regardless of
the findings of each report, this alert notifies the caseworker to initiate an
investigation when a third report is received.

Since many states are in different phases of SACWIS development, their
systems currently support to varying degrees a variety of child welfare and
administrative components (see table 3). According to HHS, while the
components represented in table 3 are required for a state’s SACWIS to be
considered compliant with federal guidance—either through an interface
or built within the system—some of the subcomponents, such as a
function that helps caseworkers manage their caseloads, are optional.
HHS has encouraged states to automate as many functions as possible in
the SACWIS in an effort to cut down on the additional paperwork or
duplicative steps inherent in manual data collection. One of these services,
tracking independent living, is becoming more important for states as HHS
decides how to implement the Foster Care Independence Act of 1999 and
considers the development of the NYTD.9 Some states have already started
collecting data on older youth and the services they receive. Currently,
27 states reported in our survey that they are at some stage of using their
SACWIS to track independent living services, and an additional 14 states
plan to include this component in their system in preparation for the



8
 Although the Iowa state officials described their SACWIS as including the child abuse and
neglect system, HHS commented on a draft of this report that it does not view the child
abuse and neglect system as part of the state’s SACWIS. However, HHS said that the state
has met the SACWIS requirement in this area by building an interface between the two
systems.
9
 The Foster Care Independence Act of 1999 increased federal support to states for
independent living programs—programs designed to assist youth who are identified as
likely to remain in foster care until age 18. Independent living services can include
education or training necessary for the youth to obtain employment.




Page 13                                                        GAO-03-809 Child Welfare
requirements. However, 21 of the 46 states reporting to our survey that
they are developing or operating a SACWIS reported that they would have
to make substantial changes to their SACWIS in order to capture this
information.

Table 3: Selected SACWIS Child Welfare and Administrative Services

                                                  Fully or partially                             Planned
    Service                                 operational in SACWIS                            for SACWIS
    Child Welfare Services
                        a
    Child protection                                                 38                                 5
                            b
    Out of home care                                                 35                                 8
    Adoption                                                         34                                 9
    Independent living                                               27                                14
                                        c
    Intensive home-based services                                    27                                13
    Administrative Services
    Workload management                                              32                                 8
    IV-E eligibilityd                                                29                                14
    Foster care maintenance
    payments                                                         28                                14
    Adoption assistance payments                                     25                                17
    Contract provider payment                                        24                                15
Source: GAO survey.

Note: Based on responses from 46 states developing or operating a SACWIS. The rows for the
columns “fully or partially operational” and “planned” do not add to 46 because the respondents may
have answered “not supported,” “don’t know,” or “no answer.”
a
    Child protection includes services such as intake and screening, investigation, and disposition.
b
    Out of home care includes things such as foster care, group homes, and residential placement.
c
    Intensive home-based services include efforts to avoid placing a child in foster care.
d
    IV-E funding is available for foster care, adoption, and independent living services.


To assist with the design of their SACWIS, states relied on a number of
different participants including internal users, such as caseworkers and
managers, information technology (IT) staff, and contractors. Most states
found these participants to be extremely or very helpful in the process
(see table 4). In Oklahoma, for example, 150 child welfare staff from the
field worked closely with the contractor in intensive work group sessions
to design and test the system. To complement the caseworkers’ knowledge
of child welfare practice, 43 states relied on IT staff. In Colorado, for
example, IT staff said that during SACWIS design and development, they
shared office space with program staff that had been assigned to help with



Page 14                                                                     GAO-03-809 Child Welfare
SACWIS development. This co-location of staff aided in the exchange of
information pertaining to the development of the system. Finally, 42 states
reported that they hired private contractors to conduct a large part of
SACWIS design and development. The contractors helped states meet
federal requirements, designed the system with state specific options,
wrote the necessary software, tested and implemented the system, and
trained users.

Table 4: Commonly Used SACWIS Development Participants and Their Level of
Helpfulness

 SACWIS development            Number of states       Number of states rating participant
 participant                   using participant           as extremely or very helpful
 Internal users                                 44                                     41
 IT staff                                       43                                     31
 Private contractors                            42                                     37
Source: GAO survey.

Note: Based on responses from 46 states developing or operating a SACWIS.


At the time of our review, HHS reported that 4 states were not pursuing
SACWIS development and most of these states reported various reasons in
our survey for not developing a system. In Hawaii, the child welfare
agency chose not to pursue SACWIS because it already had a statewide
system in place that it believed was adequately meeting its needs and
which was collecting and reporting federal child welfare data. After an
attempt to develop a system, North Carolina cancelled its efforts because
it could not build consensus across its 100 counties on the design of a
uniform system. On our site visit to North Carolina, child welfare officials
reported that they are currently working on a statewide information
system that will encompass a number of social services, such as food
stamps and mental health services, but an HHS official reported that North
Carolina is not seeking federal SACWIS funding to support the
development of this system. Vermont officials reported that they did not
pursue SACWIS because the legislature declined to provide the matching
state funds. In retrospect, they believe that the choice not to develop
SACWIS was best for the state because they found the SACWIS
requirements too restrictive to enable the state to design a system to meet
its needs. Officials said that the state would not use a number of the
required SACWIS components, such as developing all the required
electronic links to other agencies’ systems, especially since the state has a
small child welfare population. Another state—North Dakota—did not
report in our survey the reason for stopping SACWIS development;



Page 15                                                           GAO-03-809 Child Welfare
                         however, HHS officials reported that the state had attempted to develop a
                         SACWIS, but faced a variety of problems, such as receiving state funding.


States Accrue Benefits   While most state child welfare agency officials said they recognize the
from Using SACWIS, but   benefits the state will achieve by developing SACWIS, such as enhancing
                         their ability to track the whereabouts of foster children, 31 state agencies
Several Issues Create    lag behind the time frames they set for completion, with 26 states
Delays in Completing     reporting delays ranging from 2 months to 8 years. State officials reported
States’ Systems          in our survey and during site visits that SACWIS has contributed to more
                         efficient and effective agency functioning, which can improve states’
                         capabilities to manage their child welfare cases, including keeping track of
                         where the children are living and the services they are receiving. Child
                         welfare officials in Colorado reported that automation has improved
                         agency functioning by making child welfare case information available
                         statewide, which is especially helpful when families move from one county
                         to another. In Oklahoma, caseworkers and state officials noted that they
                         believe their children are safer since the implementation of SACWIS
                         simply because the information on the children is easily accessible to the
                         caseworkers and their supervisors. According to survey results, automated
                         systems provided easier access to data and allowed caseworkers to better
                         monitor children in their care, which may contribute to additional child
                         welfare and administrative benefits, such as decreased incidences of child
                         abuse and neglect, shortened length of time to achieve adoption,
                         timeliness of payments to foster families, and timeliness of payments to
                         foster facilities (see table 5). New Jersey reported in our survey that its
                         goal in developing a SACWIS is to integrate the more than 40 stand-alone
                         systems that currently capture information on the children served by their
                         child welfare agency. By pulling all of these systems together into a
                         uniform SACWIS, the state hopes to improve the recording of casework
                         activities in a timely manner and to develop a tool to better target
                         resources and services. Effectively integrating these systems will require
                         the state to use a disciplined IT management approach that includes
                         (1) detailed analyses of users’ needs and requirements, (2) a clearly
                         defined strategy for addressing information needs, and (3) sufficient
                         technical expertise and resources to support the effort.




                         Page 16                                              GAO-03-809 Child Welfare
Table 5: State-Reported Benefits of SACWIS Development

                                                        Number of states identifying SACWIS
                                                            as extremely or very effective in
 Child welfare benefits                                                   achieving measure
 Decreased incidence of child abuse and
 neglect                                                                                             25
 Shortened length of time to achieve adoption                                                        24
 Decreased recurrence of child maltreatment                                                          23
 Shortened length of time to achieve
 reunification                                                                                       22
 Management benefits
 Timeliness of payment to foster families                                                            36
 Timeliness of payment to foster facilities                                                          34
 Overall case management for children and
 families                                                                                            33
 Timeliness of child abuse and neglect
 investigations                                                                                      33
Source: GAO survey.

Note: Based on responses from 46 states developing or operating SACWIS. States not included
answered “moderately effective,” “somewhat effective,” “not at all effective,” “not a state goal,”
“system doesn’t support,” “don’t know,” or “no answer.”


Despite the benefits that many states have accrued with SACWIS, 31 states
reported in our survey that they have been delayed in system completion
beyond their initial deadline and identified a number of challenges that
have led to the delay (see table 6).10 Some of the common difficulties states
reported in developing SACWIS included receiving state funding approval,
reaching internal agreement on system development, and creating a
system that reflects child welfare work processes and is user friendly (see
table 7).




10
  Twelve of the 46 states reporting that they are developing or operating a SACWIS
reported that they have not experienced delays in developing their systems. In response to
the length of the delays reported by 26 states in our survey, ACF commented on a draft of
this report that these states may be using different definitions in defining their delays.
However, ACF did not provide further information on how the delays represented in this
report differ from its perception of states’ experiences. In our survey, we asked states to
report on the delays that exceeded the time line outlined in their initial APD.




Page 17                                                                 GAO-03-809 Child Welfare
Table 6: Number of Months States Delayed in SACWIS Development

    State                                                          Length of delay in monthsa
    Alabama                                                                                        36
    Arkansas                                                                                        6
    California                                                                                     36
    Colorado                                                                                       26
    Connecticut                                                                                    96
    District of Columbia                                                                           36
    Georgia                                                                                        25
    Idaho                                                                                          21
    Illinois                                                                                       79
    Indiana                                                                                         6
    Kansas                                                                                         72
    Louisiana                                                                                      12
    Maryland                                                                                       12
    Michigan                                                                                       26
    Minnesota                                                                                      12
    Mississippi                                                                                    12
    New Jersey                                                                                     42
    New Mexico                                                                                      3
    Ohio                                                                                           36
    Oregon                                                                                         70
    Rhode Island                                                                                   14
    South Carolina                                                                                 47
    Tennessee                                                                                      36
    Utah                                                                                           48
    Virginia                                                                                        2
    Washington                                                                                     36
Source: GAO survey.

Note: While 31 states reported in the survey that they have experienced a delay in SACWIS
development, only 26 states reported the length of their delay. The survey was issued in October
2002 and completed by states as late as December 2002.
a
States were asked to report the number of months the delays exceeded the time line outlined in their
APD.




Page 18                                                               GAO-03-809 Child Welfare
Table 7: Common SACWIS Development Challenges

                                                            Number of states experiencing at
                                                                                           a
    SACWIS development challenges                                     least some challenge
    Receiving state funding approval                                                             42
    Reaching internal agreement on system
    development                                                                                  41
    Creating a system that reflects work
    processes                                                                                    40
    Creating a user friendly system                                                              39
    Insufficient state funding allocation                                                        32
    Securing contractors with knowledge of child
    welfare                                                                                      32
Source: GAO survey.

Note: Based on responses from 46 states developing or operating a SACWIS.
a
 States were asked the extent to which certain factors were a challenge in developing SACWIS using
the following scale: very great, great, moderate, some, or no challenge. The number of states
reported in our analysis of challenges represents the total number of states reporting any extent the
factor posed a challenge. States not included answered “no challenge,” “have not yet reached this
stage,” “don’t know,” or “no answer.”


Forty-two states reported challenges receiving funding approval, and
32 states reported that insufficient state funding allocations for SACWIS
development were a challenge in developing a comprehensive system. For
example, Vermont officials reported that the state legislature declined to
provide the matching state funds needed to secure federal funding for
SACWIS. As a result, the state could not pursue development. In addition
to the development challenges reported in our survey, 2 of the 5 states we
visited reported that insufficient funding affected ongoing SACWIS
maintenance. In Colorado, state agencies have received a series of budget
cuts, which child welfare officials report have impacted their ability to
respond to child welfare caseworkers’ needs for system improvements. In
Iowa, child welfare officials reported that the state legislature
appropriated $17,000 for state fiscal year 2002 for all child welfare
automated systems activity, which they noted was an insufficient amount
to maintain and upgrade systems as well as to pay staff. They reported
that, as a result, the child welfare agency provided the information
systems department with approximately $1 million from other parts of the
agency’s budget.

Despite user involvement in system design, some states still faced
challenges trying to reach internal agreement among agency officials and
caseworkers on the design of a system, resulting in a delay in
development. In New York—a state where the counties are responsible for


Page 19                                                                GAO-03-809 Child Welfare
administering child welfare services—the development of SACWIS was
stalled when significant frustration with the system’s design led
Commissioners from five large counties and New York City to request that
the state stop SACWIS development until a reassessment of the design and
plans for the implementation of the system was completed. After a
thorough evaluation of the project, the state made changes to the project
plan and developed statewide work groups to ensure all counties were in
agreement with the system design. In addition, they hired a contractor to
monitor system development and ensure that all users’ requirements are
seriously considered.

Similarly, despite states’ heavy reliance on contractors, many reported that
securing contractors with knowledge of child welfare practice was a
challenge for timely SACWIS development. Contractors are hired by the
state for their system development knowledge but often are unfamiliar
with child welfare policies and practices, especially since they vary from
state to state. Officials in Colorado, for example, said they encountered
difficulties with their contractors because of high turnover among the
contractor staff and their lack of knowledge of child welfare policies. A
contractor who has worked with 7 states to develop their SACWIS
reported that contractors are asked to learn the child welfare business
practices of a state in a short amount of time and that states cannot devote
many resources, such as caseworkers, to help in the design process
because caseworkers need to devote their time to providing services to
children and families. Therefore, contractors often have to acquire
knowledge on their own.

Many states reported that creating a system that reflects child welfare
work processes and is user friendly was a challenge in developing
SACWIS. These issues were also identified in the federal reviews of states’
SACWIS. For example, one state explained in the SACWIS review that it
had designed a system to meet the caseworkers’ needs and reflect the
nature of the child welfare work processes by developing a system that
required events to be documented as they occurred. However, this design
limited the SACWIS’s functionality because it did not allow the
caseworkers to go back and enter information after an event happened.
The state explained that caseworkers do not use the system in real time,
but provide services to the children and families and then record the
information in the system. The state had to redesign the system to correct
for this design flaw. In addition, the 14 states reporting that they have
adapted a system from another state have experienced some challenge
modifying the systems to reflect their work processes. While HHS advises
states to consider adapting another state’s system if it requires few


Page 20                                              GAO-03-809 Child Welfare
                            changes, states report that they and their contractors were not always
                            aware of the changes that would need to be made to adapt a system.
                            Although Colorado and New York decided to modify another state’s
                            SACWIS instead of designing a new system, child welfare officials in these
                            states reported that adapting a SACWIS from another state has created
                            more problems than anticipated. Colorado and New York adapted systems
                            from state-administered child welfare agencies, which required extensive
                            modifications to meet their needs as county-administered states. For
                            example, Colorado needed a system that supported its administrative
                            structure and could handle a larger number of cases.


HHS Provides Some           HHS has assisted states in a variety of ways in developing and completing
Assistance to Help States   their SACWIS.11 As a part of its regulatory responsibilities, HHS must
Meet SACWIS                 review, assess, and inspect the planning, design, development, installation,
                            and operation of SACWIS. In addition to reviewing and monitoring states’
Requirements                APDs, HHS conducts on-site SACWIS reviews to comply with these
                            responsibilities. HHS officials told us that these reviews are a detailed and
                            thorough assessment of state systems to ensure the systems’ compliance
                            with SACWIS requirements. In addition, officials reported that they
                            provide verbal technical assistance during the on-site review to help states
                            that do not fully conform with the applicable regulations and policies. At
                            the time of our review, HHS had conducted 26 SACWIS reviews—5 of
                            which were determined as meeting all the requirements and classified as
                            complete. HHS officials told us that since states have the flexibility to
                            build a SACWIS that meets their needs, a large portion of the formal
                            reviews concentrate on ensuring that the systems conform to state
                            business practices. For example, while SACWIS regulations require that a
                            state report all AFCARS data from their SACWIS, one state HHS reviewed
                            relied on a separate state system to report data on the children served by
                            the juvenile justice agency who are eligible for IV-E foster care funds. The
                            state proved it had developed an automated process to merge data from
                            both systems to compile a single AFCARS report that included children
                            captured in both their SACWIS and juvenile justice systems. Therefore,



                            11
                             With regard to the budget difficulties that states reported facing, since 1994 the federal
                            government has made a commitment to help states develop and maintain their SACWIS by
                            matching 75 percent of states’ development funds through 1997 and providing an ongoing
                            match of 50 percent of state funding for the development and maintenance of their
                            systems. However, since the states’ legislatures must make the initial commitment to fund
                            SACWIS, the federal government cannot assist state child welfare agencies with this
                            challenge.




                            Page 21                                                         GAO-03-809 Child Welfare
HHS recognized that this process best met the state’s needs and
determined the SACWIS to be complete and meeting all requirements.

Few systems have been determined complete after an on-site review
because of unresolved issues, such as not being able to build links to other
state information systems or not implementing certain eligibility
determination functions. To help states address some of these
development challenges, the SACWIS review team provides the state with
recommendations for complying with SACWIS requirements. For example,
HHS observed during a review in one state that the SACWIS was available
statewide, but information collected in one county was not available to
caseworkers in other counties. The federal officials offered
recommendations to the state to meet the SACWIS requirement that all
information be available statewide. In addition, HHS officials reported that
once the draft report with the results of the SACWIS review are
completed, federal staff schedule a conference call with the state officials
to walk through the system’s deficiencies and offer guidance on how the
state can move forward.

HHS facilitates the sharing of information between states developing
SACWIS through an automated system users group that allows state and
federal officials to exchange information, ideas, and concerns. According
to some state and HHS officials, the trust level at these meetings is very
high, which promotes open discussions and also creates an atmosphere
for informal dialogue with HHS. The systems users group developed out of
another active group—the child welfare users group—when HHS solicited
state representatives to help HHS define a model child welfare information
system, which was later used as the basis for the SACWIS functional
requirements after the passage of the 1993 legislation authorizing
enhanced federal funding. State officials in Iowa and New York reported
that the systems users group continues to play an important role in
providing a forum for the honest exchange of information on SACWIS
development. For example, child welfare and technical officials in New
York said that the systems users group has been very beneficial because
they have learned from other states’ positive and negative experiences in
developing SACWIS, as well as the experiences unique to states with
county-administered agencies. In addition to the users group, HHS officials
also sponsor a listserv—an electronic mailing list—that allows state
officials to exchange information, and a monthly conference call with state
information technology directors. Iowa child welfare information




Page 22                                              GAO-03-809 Child Welfare
technology officials said that they find the monthly SACWIS telephone
conference call helpful because project managers discuss issues such as
promising practices and new regulations.12

Technical assistance for SACWIS development is also available to states
through the National Resource Center for Information Technology in Child
Welfare (Resource Center). According to survey results, 9 states said they
used the Resource Center for assistance in developing SACWIS and
14 states reported using it for help with SACWIS maintenance and
improvements. According to Resource Center officials, they assist states
with SACWIS development by helping states understand the technology
that is available for use, providing information on the automation of child
welfare work and converting data, and reviewing the APD documentation.
For example, the Resource Center offered technical assistance to
Pennsylvania to help the state decide if it should continue development of
its current SACWIS, abandon the SACWIS project and allow the counties
to operate individual systems, or design a different SACWIS. The Resource
Center evaluated the current SACWIS to determine if it could capture
information based on the SACWIS regulations and if it was user friendly
for the caseworker. Following the Resource Center’s analysis,
Pennsylvania decided to discontinue the existing SACWIS and develop a
new SACWIS. When the Resource Center opened in 1999—5 years after
many states started developing SACWIS—staff were not very familiar with
many of the efforts states made during development. In an attempt to
remedy this lack of knowledge on states’ issues developing SACWIS,
Resource Center staff participated in some of the on-site SACWIS reviews
conducted by HHS. Both HHS and Resource Center officials believe this
exposure to the SACWIS systems enhanced the availability of technical
assistance resources and knowledge available to the states.




12
 In commenting on a draft of this report, HHS indicated that a Web resource is available to
states interested in learning about other states’ efforts to develop human services—child
welfare, food stamps, Temporary Assistance to Needy Families, child care, and child
support enforcement—information systems at http://www.acf.hhs.gov/nhsitrc.




Page 23                                                        GAO-03-809 Child Welfare
                         Several factors affect states’ ability to collect and report reliable13 data on
Several Factors Affect   children served by state child welfare agencies, and some problems exist,
the States’ Ability to   such as a lack of clear and documented guidance, with HHS’s oversight
                         and technical assistance. Almost all of the states responding to our survey
Ensure Reliable Data     reported that insufficient caseworker training and inaccurate and
on Children’s            incomplete data entry affect the quality of the data reported to HHS.14 In
                         addition, 36 of the 5015 states that responded to our survey reported that
Experiences, and         technical challenges, such as matching their state data element definitions
Some of HHS’s            to HHS’s data categories, affected the quality of the data that they report to
Oversight and            the federal government. For example, North Carolina officials told us that
                         while state policy mandates that they count every location in which a child
Assistance Is            resides, including hospital stays, AFCARS regulations say that hospital
Problematic              stays and other short-term placements should not be included in the count
                         of foster care placements. In cases where state policy differs from federal
                         policy, state officials must carefully re-format their data in order to meet
                         federal reporting requirements. Similarly, during assessments of 6 states’
                         compliance with AFCARS reporting standards, HHS found that these
                         issues affect data reliability. Despite the assistance that HHS offers to
                         states, such as testing state data quality and providing the results to states
                         to aid them in resubmitting data, states report ongoing challenges
                         receiving clear and documented guidance and accessing technical
                         assistance.




                         13
                          Data are reliable when they are complete and accurate. A subcategory of accuracy is
                         consistency. Consistency refers to the need to obtain and use data that are clear and well-
                         defined enough to yield similar results in similar analysis. See U.S. General Accounting
                         Office, Assessing the Reliability of Computer-Processed Data, GAO-02-15G (Washington,
                         D.C.: Sept. 2002).
                         14
                          States were asked the extent to which certain problems may decrease the quality of the
                         data submitted to AFCARS and NCANDS using the following scale: very great, great,
                         moderate, some, and no affect.
                         15
                          The analysis of survey responses about reporting data to HHS is based on responses from
                         49 states and the District of Columbia. All states, regardless of SACWIS development, were
                         asked to complete these questions.




                         Page 24                                                         GAO-03-809 Child Welfare
Insufficient Caseworker     Almost every state responding to our survey and all the states we visited
Training and Inaccurate     reported that insufficient training for caseworkers and inaccurate and
and Incomplete Data Entry   incomplete data entry affect the quality of the data reported to AFCARS
                            and NCANDS (see fig. 1). Although most states reported these as separate
Are the Most Common         factors, HHS and the states we visited found that insufficient training and
Factors That Affect Data    inaccurate and incomplete data entry are often linked. For example, in
Reliability                 official reviews of states’ information systems capability to capture data
                            and report them to AFCARS, HHS advised states to offer additional
                            training to caseworkers on several AFCARS data elements, such as
                            recording the reasons for a child leaving foster care, to improve the
                            accuracy of the data submitted. Similarly, Oklahoma reported that the
                            state found that caseworkers were misinterpreting reports of policy
                            violations by foster parents and inaccurately recording them as abuse or
                            neglect allegations. However, state officials told us that training is typically
                            one of the first programs cut when states face tight budget restrictions.
                            For example, Iowa officials told us that training has been significantly
                            reduced in recent years because of budget cuts and new workers may wait
                            2 to 3 months before being trained how to enter data appropriately into
                            their SACWIS.




                            Page 25                                                GAO-03-809 Child Welfare
Figure 1: Most Common Caseworker Issues That Affect Data Quality

States reporting at least some affect in data quality
50       49                    49
                                                        46
                  45
                                                             43
                                       42
                              15
40       18
                 12
                                      11                17   12


30




20                            34
                 33
          31                          31                29   31


10




 0

        Inaccurate data       Insufficient          Incomplete data
        entry by              training for          entry by
        caseworkers           caseworkers           caseworkers



                  AFCARS

                  NCANDS

Source: GAO survey.

Notes: Based on responses from 50 states.

The results reported in the figure are a sum of the states that reported the issue had a very great
affect, great affect, moderate affect, or some affect on the quality of state data submitted to HHS.
Very great and great affect responses are represented in the top section of each bar. Moderate and
some affect responses are represented in the bottom section of each bar. States not included
answered “no affect,” “don’t know,” or “no answer.”


Inaccurate and incomplete data entry can also result from a number of
other factors, such as caseworkers’ hesitation to ask families for sensitive
information. For example, caseworkers in Oklahoma reported that they
did not feel comfortable asking if a child’s mother was married at the time
of birth or if a child is of Hispanic origin—both of which are required
AFCARS data elements. In commenting on a draft of this report, Oklahoma
added that caseworkers did not understand why the data elements were
required and how the federal government used the information. In
addition, Iowa state officials said that caseworkers may guess the racial
backgrounds of children in their care or record them as unknown,
especially when children come from mixed racial backgrounds, rather
than asking the family for the information. HHS noted similar issues in



Page 26                                                                GAO-03-809 Child Welfare
5 states that have had an AFCARS review.16 Caseworkers were
inaccurately recording a child’s race as “unable to determine” even though
this option should be selected only if the child’s parents or relatives
cannot provide the information, such as when a child is abandoned.17

Caseworkers, supervisors, and managers in the 5 states we visited
reported that additional factors, such as difficulties balancing data entry
with the time that they spend with the families and children, contributed
to inaccurate or incomplete data entry. In addition, our recent work on
caseworker recruitment and retention found that caseworkers struggle to
balance the time they spend with children and data entry, and reportedly
spend at least 50 percent of their time documenting case records.18
Supervisors in Iowa explained that since caseworkers are responsible for
ensuring that children and their families receive the services they need, the
caseworkers tend to initially limit data entry to the information that is
necessary to ensure timely payment to foster care providers, and complete
all other data elements when the caseworkers have time. In addition,
caseworkers in Colorado said that they are between 30 and 60 days behind
in their data entry, so the information in the automated system may not
accurately reflect the current circumstances of children in care. The
caseworkers reported that they tend to concentrate only on entering data
that will allow them to open a case in their SACWIS. HHS’s Inspector
General recently issued a report in which more than two-thirds of the
states reported that caseworkers’ workloads, turnover, a lack of training,
and untimely and incomplete data entry affected the reporting of AFCARS
data.19



16
 We reviewed AFCARS reports from 6 of the 8 states assessed by HHS—Arkansas,
Connecticut, New Mexico, Texas, Vermont, and Wyoming. HHS conducted reviews in
Delaware and West Virginia after we completed our analysis.
17
  In commenting on a draft of this report, ACF said that the finding from the AFCARS
reviews indicates that information is often defaulted to the response “unable to determine”
in order for the element not to fail the missing data standard, not that workers are
recording “unknown”; however, the report findings we used in this analysis instruct states
to fix the defaults and address caseworker practice by enhancing training on the correct
use of “unable to determine” when noting a child’s race.
18
 See U.S. General Accounting Office, Child Welfare: HHS Could Play a Greater Role in
Helping Child Welfare Agencies Recruit and Retain Staff, GAO-03-357 (Washington, D.C.:
Mar. 31, 2003).
19
 Department of Health and Human Services, Office of Inspector General, Adoption and
Foster Care Analysis and Reporting System (AFCARS): Challenges and Limitations,
OEI-07-01-00660 (Washington, D.C.: Mar. 2003).




Page 27                                                         GAO-03-809 Child Welfare
Technical Challenges, such   In addition to data quality being affected by caseworker issues, many
as Matching State            states experienced technical challenges reporting their data to HHS. The
Definitions to Federal       problems reported by states are typically a result of challenges associated
                             with data “mapping”—matching state data elements to the federal data
Definitions, Affect Data     elements. For example, 36 states reported in our survey that matching
Reliability                  their state-defined data to HHS’s definitions affected the quality of the data
                             reported to NCANDS and AFCARS. Similarly, 24 states reported that
                             matching the more detailed data options available in their states’
                             information systems to the federal data elements affected the quality of
                             the data reported to NCANDS. Twenty-nine states reported that this issue
                             created challenges in reporting data to AFCARS. For example, following
                             an AFCARS assessment, HHS instructed a state that collects detailed
                             information on children’s disabilities, such as Downs Syndrome, Attention
                             Deficit Disorder, and eating disorders, to map the information to the more
                             limited options in AFCARS, such as mental retardation and emotionally
                             disturbed. The Inspector General’s report found that states faced similar
                             challenges mapping their data to meet the AFCARS reporting
                             requirements.

                             In many cases, states have to balance state policy with federal
                             requirements to ensure that they are reporting accurate data to AFCARS
                             and NCANDS, but are not contradicting their state policies. For example,
                             Texas officials reported that although the findings of their AFCARS review
                             instructed them to modify their SACWIS to collect, map, and extract data
                             on guardianship placements, the state does not support guardianship
                             arrangements.20 In addition, a recent report from the Child Welfare League
                             of America (CWLA) found that when reporting the number of times
                             children move from one foster care placement to another, states varied in
                             the type of placements included in that count.21 For example, 29 percent of
                             the states responding to CWLA’s survey included respite,22 25 percent
                             included runaways, and 16 percent included trial home visits when
                             reporting the number of placements a child had during the AFCARS report
                             period. According to federal guidance, the “number of placements”
                             element is meant to gather information on the number of times the child


                             20
                              Guardianship arrangements occur when permanent legal custody of a child is awarded to
                             an individual, such as a relative, but the child is not legally adopted.
                             21
                              Child Welfare League of America. National Working Group Highlights, “Placement
                             Stability Measure and Diverse Out-of-Home Care Populations” (Washington, D.C., Apr.
                             2002).
                             22
                              Respite care provides temporary childcare for children away from their caretakers.




                             Page 28                                                       GAO-03-809 Child Welfare
                            welfare agency found it necessary to move a child while in foster care and
                            that by including runaways or trial home visits, a state is inflating the
                            number of moves a child experienced. However, North Carolina officials
                            told us that although the federal definition for placements instructs states
                            not to include such stays when counting the number of children’s foster
                            care placements, the state instructs them to count each time a child is
                            sleeping in a different place as a new placement. The Inspector General
                            reported that the placement definitions were the most commonly cited
                            source of confusion among the states surveyed.23

                            In addition to the challenges reported in our survey, HHS reported that
                            transferring data from older data systems into SACWIS affects the quality
                            of the data reported to AFCARS and NCANDS. HHS officials reported that
                            they have observed that states experience the biggest change in data
                            quality when they begin reporting from their SACWIS. In general, the first
                            data submissions are of low quality because of the time it takes states to
                            transfer data or the system re-sets the information for data elements. For
                            example, in 1 state, 65 percent of the records reviewed by HHS during an
                            AFCARS assessment recorded the date the children were removed from
                            their homes as July 28, 1997—the date the SACWIS came on-line; however,
                            the actual dates of removal for these children ranged from 1988 to 1997.


Although HHS Has Taken      HHS provides technical assistance for AFCARS and NCANDS reporting
Steps to Help States        through a number of resources. HHS officials in the central office and
Improve Their Data, Some    NCANDS contractor staff serve as the points of contact for states to ask
                            questions and seek guidance on reporting child welfare data. HHS officials
Problems with Its Efforts   reported that assistance is offered in a number of ways, including
Exist                       telephone and e-mail communication. The officials in 3 of the 5 states that
                            we visited said that the one-on-one focused technical assistance was
                            useful when provided in a timely fashion. Most state officials found the
                            NCANDS data easier to report, in part because more people were available
                            for consultation and they were more accessible and responsive. For
                            example, states have access to four NCANDS specialists and staff in the
                            contractor’s central office when they need assistance reporting child
                            abuse and neglect information. However, some of the states we visited




                            23
                             Although the findings from the Inspector General’s report and our study are more recent,
                            ACF commented on a draft of this report that it issued policy clarifications regarding
                            placement information on July 5, 2002.




                            Page 29                                                       GAO-03-809 Child Welfare
reported that only one or two staff in HHS’s central office are available to
assist with AFCARS reporting.

In addition, the Resource Center offers states assistance with improving
data quality; however, Resource Center staff reported that the assistance is
geared more towards improving the limited data used in the federal review
process to monitor states’ compliance with child welfare laws and federal
outcome measures—CFSR—rather than all the data reported to HHS. The
Resource Center also sponsors an annual information technology
conference during which sessions covering all data-related issues are held,
including practices for ensuring data quality and outcome evaluation in
child welfare.

In conjunction with the national data conference, the HHS officials and the
contractors that operate NCANDS hold an annual technical assistance
meeting for states to share ideas with one another, discuss data elements
that pose difficulties, and explore ways to address these problems. For
example, at a recent technical assistance meeting, approximately 43 state
representatives attended sessions on preparing the calendar year 2002
NCANDS data submissions and received a detailed explanation of how the
NCANDS staff test states’ data submissions for quality. In addition, an
NCANDS state advisory group meets annually to talk with HHS officials
about NCANDS data and their experiences reporting data. From these
meetings, the state advisory group proposes changes or improvements to
NCANDS. HHS and state officials reported that this partnership has helped
ease some of the challenges in reporting child abuse and neglect data.

In addition to the direct assistance through consultation with HHS officials
and the Resource Center, HHS has made available to states the software it
uses to examine states’ AFCARS and NCANDS submissions for
inconsistencies and invalid data. Officials in all the states we visited said
that they regularly use this software, and an HHS official said that nearly
every state has used the software at least once. When the data are
submitted to HHS, they are run through the same software, and HHS
notifies the states of areas where data are missing or inconsistent and
allows the states to resubmit the data after errors are corrected. For
example, HHS officials said that they worked with one state that was
trying to determine the source of data errors in reporting to AFCARS the
race or ethnicity of children in their care. The state was not able to
determine the source of the problem, so an HHS official examined the
state’s submissions and helped correct the data errors. The officials
reported that these tests help them to identify some data quality errors,
such as missing data, and said that they believe that, in general, data have


Page 30                                               GAO-03-809 Child Welfare
improved in recent years. However, they indicated that the tests cannot
pinpoint the underlying problems contributing to these errors.
Furthermore, one official reported that no specific efforts have been
conducted to track the individual data elements and, therefore, HHS
cannot report on how data quality has changed over time. The results of
these quality tests had been the basis for penalties levied against states
that submitted low quality AFCARS data before the penalties were
rescinded. HHS officials reported that the penalties served as an effective
motivation to states to correct their data. Although HHS was not able to
report how the lack of penalties might be affecting recent data quality, an
official reported that the agency plans to conduct this analysis in the
future.

In an attempt to help states comply with the reporting standards and
address some of the factors that contribute to data quality problems, HHS
performs comprehensive reviews of state information systems’ ability to
capture AFCARS data to identify problems associated with data collection
and reporting, and to ensure that the information in the automated system
correctly reflects children’s experiences in care. The assessments include
a technical review of the states’ computer code, a comparison of the data
from selected cases available in the information system to the case files,
and an improvement plan to resolve any errors. In addition, HHS officials
offer guidance to the states on improvements that can be made to the
information system and changes to program code used to report the
AFCARS data. HHS conducted pilot reviews in eight states between 1996
and 2000. By March 2003, HHS had conducted eight official reviews—even
though states began reporting to AFCARS in 1995. According to results
from six of the eight official AFCARS assessments we reviewed, no state
met the reporting requirements for all AFCARS data elements. Table 8
shows a selection of the data elements and the states’ ratings. The
problems noted in the reviews are similar to those we heard from states
responding to our survey and those we visited. For example, most states




Page 31                                              GAO-03-809 Child Welfare
                                             received ratings of 2 or 3, indicating technical and/or data entry errors that
                                             affect the AFCARS data quality.24

Table 8: Selected AFCARS Elements and Six States’ Levels of Compliance in Meeting Reporting Requirements

                                                                                                                         Rating factora
Data Elements                                                                                                              (# of states)
Foster Care
Child race                                                                                                                  2 (4 states)
(American Indian or Alaska native, Asian, Black or African American, Native Hawaiian or                                      3 (1 state)
other Pacific Islander, White, unable to determine)                                                                          4 (1 state)
Has the child been diagnosed with a disability?                                                                             2 (6 states)
(yes, no, not yet determined)
Mental retardation, visually/hearing impaired, physically disabled, emotionally disturbed,                                  2 (5 states)
other diagnosed conditionb                                                                                                   3 (1 state)
(does not apply, applies)
Has child ever been adopted?                                                                                                 1 (1 state)
(yes, no, unable to determine)                                                                                              2 (3 states)
                                                                                                                            3 (2 states)
                                       c
Total number of removals from home                                                                                          2 (3 states)
                                                                                                                            3 (2 states)
                                                                                                                             4 (1 state)
Date of discharge from previous episodec                                                                                    2 (3 states)
                                                                                                                            3 (2 states)
                                                                                                                             4 (1 state)
Date of latest removalc                                                                                                     2 (3 states)
                                                                                                                            3 (3 states)
Date of placement in current setting                                                                                        2 (5 states)
                                                                                                                             4 (1 state)
                                                       c
Number of previous placement settings in this episode                                                                       2 (6 states)
Current placement setting                                                                                                   2 (4 states)
(pre-adoptive home, foster family home-relative, foster family home-nonrelative, group                                       3 (1 state)
home, institution, supervised independent living, runaway, trial home visit)                                                 4 (1 state)
Out of state placement (yes, no)                                                                                            2 (4 states)
                                                                                                                             3 (1 state)
                                                                                                                             4 (1 state)




                                             24
                                              A rating of 2 indicates that the state’s information system does not fully meet the
                                             requirements for AFCARS reporting, whereas if a state receives a 3, the information system
                                             requirements for AFCARS reporting are in place, but there are data entry problems
                                             affecting the quality of data. According to an HHS official, data elements that have a
                                             combination of technical and data entry problems are rated as 2 until the technical issues
                                             are resolved. HHS will then rate the element as a 3 until the data entry practices are
                                             changed. A state receives a rating of 4 if in compliance with the AFCARS requirements or a
                                             1 if the requirement is not addressed in the state’s information system.




                                             Page 32                                                        GAO-03-809 Child Welfare
                                                                                                                                                       a
                                                                                                                                      Rating factor
 Data Elements                                                                                                                          (# of states)
 Most recent case plan goal                                                                                                               1 (1 state)
 (reunify with parents or principal care takers, live with other relative(s), adoption, long term                                        2 (5 states)
 foster care, emancipation, guardianship, case plan goal not yet established)
                   c
 Date of discharge                                                                                                                       2 (2 states)
                                                                                                                                         3 (2 states)
                                                                                                                                         4 (2 states)
 Reason for dischargec                                                                                                                   2 (4 states)
 (not applicable, reunification with parent(s) or primary caretaker(s), living with other                                                 3 (1 state)
 relative(s), adoption, emancipation, guardianship, transfer to another agency, runaway,                                                  4 (1 state)
 death of child)
 Adoption
 Child race                                                                                                                              2 (4 states)
 (American Indian or Alaska native, Asian, Black or African American, Native Hawaiian or                                                  3 (1 state)
 other Pacific Islander, White, unable to determine)                                                                                      4 (1 state)
 Primary basis for determining special needs                                                                                             2 (6 states)
 (not applicable; racial/original background; age; membership in a sibling group; medical
 conditions or mental, physical, or emotional disabilities; other)
 Mental retardation, physically disabled, emotionally disturbedb                                                                         2 (5 states)
 (does not apply, applies)                                                                                                                3 (1 state)
 Visually/hearing impaired                                                                                                               2 (4 states)
 (does not apply, applies)                                                                                                               3 (2 states)
 Other diagnosed condition                                                                                                               2 (6 states)
 (does not apply, applies)
Source: HHS.

                                               Note: Analysis based on results of AFCARS reviews in 6 states. States are rated on 66 foster care
                                               elements and 37 adoption elements.
                                               a
                                                HHS rates each data element using a four-point scale: (1) the AFCARS requirement(s) has not been
                                               implemented in the information system; (2) the technical system requirements for AFCARS reporting
                                               do not fully meet the standards; (3) the technical system requirements for AFCARS reporting are in
                                               place, but there are data entry problems affecting the quality of the data; (4) all of the AFCARS
                                               requirements have been met.
                                               b
                                                   States are rated on each category. The data elements were combined for purposes of this analysis.
                                               c
                                                   AFCARS data element used in the CFSR.


                                               State officials in these 6 states reported that they found the reviews useful
                                               for improving their AFCARS data submissions. In particular, they valued
                                               the thorough review by HHS officials of the computer code states use to
                                               report the data. Some of these officials reported that if all states were
                                               reviewed, the quality of data available in AFCARS would improve
                                               tremendously. However, HHS officials reported that they are not
                                               mandated to conduct the AFCARS reviews and that priority is placed on
                                               other reviews, such as the CFSR and SACWIS reviews. In addition,
                                               officials explained that the AFCARS reviews are not conducted in states
                                               developing SACWIS until the systems are operational. HHS expects to


                                               Page 33                                                                 GAO-03-809 Child Welfare
complete approximately four reviews each year depending on available
resources and has scheduled states through 2006. Similar to the SACWIS
reviews, HHS officials offer recommendations and technical assistance to
states during the review on how they can improve the quality of the data
reported to AFCARS.

Although the states we visited appreciated some of HHS’s efforts to assist
with improving state data quality, they and most states responding to our
survey agreed that the assistance is not always consistent or easily
accessible (see fig. 2). States reported similar information to the Inspector
General—AFCARS data elements were not clearly and consistently
defined and technical assistance is effective but difficult to access.




Page 34                                               GAO-03-809 Child Welfare
Figure 2: Federal Practices That Affect Data Quality

States reporting at least some affect in data quality
50
      45

                             41
40
       17       35                                 35
                             11
                                                   6
30             9
                                     25

                                     5
20
                                                         17
                             30                    29
       28                                                4
               26
                                    20
10
                                                         13


 0

       Imprecise           Lack of clear,          Difficulty
     definitions for       documented             accessing
     required fields         guidance              technical
                                                  assistance
                                                 and guidance

              AFCARS

              NCANDS

Source: GAO survey.

Notes: Based on responses from 50 states.

The results reported in the figure are a sum of the states that reported the issue had a very great
affect, great affect, moderate affect, or some affect on the quality of state data submitted to HHS.
Very great and great affect responses are represented in the top section of each bar. Moderate and
some affect responses are represented in the bottom section of each bar. States not included
answered “no affect,” “don’t know,” or “no answer.”


The primary concerns reported by the states we visited were delays in
receiving clear, written guidance on defining and reporting certain data
elements and the lack of state input in suggesting changes to AFCARS.
Despite the written guidance available to states in the form of regulations
and an on-line policy manual, states reported that the variation in state
policies and practices make it difficult to interpret how to apply the
general guidance. As a result, states consult with HHS to ensure they are
applying the regulations appropriately. However, in commenting on a draft
of this report, officials in Oklahoma told us that a common concern among
the states is the lack of timely response from HHS when seeking guidance
on how to report data. In addition, officials in New York explained they


Page 35                                                                GAO-03-809 Child Welfare
have made it a practice to check the HHS Web site on a regular basis for
current guidance, but have not found it a useful tool, and may turn to other
states for guidance on AFCARS reporting. In commenting on a draft of this
report, HHS explained that it first refers states to its Web site for
information and believes that the available guidance addresses states’
concerns in most instances. In addition, the states that have had an
AFCARS review experienced delays in obtaining guidance on how to
proceed following the on-site review. Although they found the review to be
very helpful, some states reported that HHS officials are delayed in
responding to their questions. For example, Texas officials reported that
the state sought clarification on its improvement plan and submitted
additional questions to HHS following the review; however, when we
spoke with the state officials, they said that they had been waiting 3
months for a response on how to proceed. An HHS official told us that
since the review process is relatively new, the agency is still developing a
process to respond to the states and recognizes that it has not been
responsive to the states already reviewed. In addition, HHS is taking steps
to gather feedback from states and other users of AFCARS data to
determine how to improve the system to make the data more accurate and
useable. As a part of these efforts, HHS has published a Federal Register
notice soliciting comments and held focus group meetings at national
conferences.

The difficulties states face in receiving federal guidance and assistance, as
well as the other challenges they face in reporting data, may negatively
impact the reliability of the data available in AFCARS and NCANDS. As a
result, states are concerned that the national standards used in the CFSR
are based on unreliable data and should not be used as a basis of
comparison and potential financial penalty. The variation in states’
reporting practices may affect the validity of the measures and may place
some states at a disadvantage. For example, the CWLA and Inspector
General studies found that approximately half the states include the
juvenile justice population in their AFCARS reports, while the other states




Page 36                                               GAO-03-809 Child Welfare
do not.25 Child welfare experts and some state officials believe that the
states that include children served by the juvenile justice agency in their
AFCARS report may report a higher number of re-entries into the child
welfare system or a higher number of moves within the system when
compared to states that do not have IV-E agreements26 with their juvenile
justice systems.27 As a result, a state that includes such children in their
AFCARS report are likely to fare less favorably when compared to the
national standard than other states on two outcome measures—foster care
re-entries and stability of foster care placements—and may face financial
penalties associated with the CFSR.




25
  Federal guidance states that children who are (1) removed from their home and initially
placed in a juvenile justice facility are not to be included in AFCARS reporting; (2) in a
foster care setting who are moved to a juvenile justice facility and who are expected to be
returned to a foster care setting should continue to be included in the AFCARS reporting
population; and (3) in a foster care setting and are moved to a juvenile justice facility and
who become the responsibility of another agency should not be included in the AFCARS
reporting population. However, these studies show that the arrangements between child
welfare and juvenile justice agencies vary, which affect the population of children reported
to AFCARS. For example, some state child welfare agencies have responsibility for all
children in the juvenile justice system and include these children in their AFCARS reports,
while other states only report children who are in the custody of the juvenile justice
system, but receiving Title IV-E funding.
26
 According to HHS, state child welfare agencies can claim Title IV-E foster care
maintenance funds for eligible children for costs incurred by juvenile justice agencies that
have entered into an agreement with the child welfare agencies. Delinquent children served
by these agencies who meet the Title IV-E eligibility criteria are eligible for Title IV-E foster
care maintenance funds. States must meet all Titles IV-B and IV-E program and/or eligibility
requirements with respect to the children who are adjudicated delinquent and are receiving
Title IV-E funding or for children in foster care under the placement and care responsibility
of the child welfare agency.
27
  Children in the juvenile justice system may enter care because of behavior problems and
return home when the behavior is controlled. However, the problem may arise again and
the child could re-enter state custody. Also, the children may experience a greater number
of placements while in foster care because of their behavior problems or a “step-down”
approach that some states use to gradually decrease the level of security required to care
for the children.




Page 37                                                            GAO-03-809 Child Welfare
                           Some states are using a variety of practices to address the challenges
States Are Using           associated with developing SACWIS and improving data reliability,
Various Practices to       although no formal evaluations are available on their effectiveness. To
                           address the challenge of developing a system to meet statewide needs,
Overcome System            states relied on caseworkers and supervisors from local offices to assist in
Development                the design and testing of the system. Few states reported in our survey
                           strategies to overcome the other key challenges, such as limited funding
Challenges and             and securing knowledgeable contractors, but some states we visited have
Improve Data on            devised some useful approaches. For example, Oklahoma child welfare
Children’s                 officials—in order to maximize the limited state funding for maintaining
                           their SACWIS—reported saving $1 million each year by hiring some of the
Experiences                contractors who developed their SACWIS as permanent staff. To improve
                           data reliability, the 5 states we visited routinely review their data to
                           identify data entry errors so that managers can ensure that the missing
                           data are entered appropriately. In addition, some states reported that
                           frequent use of the data, such as publishing periodic management reports
                           detailing local offices’ performance on outcome measures, helps
                           caseworkers understand the importance of entering timely information.


States Are Primarily       To overcome development challenges, survey respondents emphasized the
Relying on SACWIS Users    importance of including system users in the various phases of completing
to Overcome Some of the    SACWIS—planning, design, development, testing, and implementation.
                           Past GAO work and other research efforts have determined similar
Challenges to Completing   approaches as best practices in building information systems.28 Forty-four
Their Systems              of the 46 states responding to our survey that they are developing or
                           operating a SACWIS indicated that they relied on internal users, such as
                           caseworkers and supervisors, in the development of their systems and 34
                           of these states said that they were extremely helpful participants. The
                           extent to which the users were involved in development differed across
                           the states. For example, in Texas, caseworkers from all of their child
                           welfare regions were recruited to provide input on design and
                           development, as well as during initial testing, pilot testing, and
                           implementation of the system. Arkansas reported establishing a committee
                           made up of users to review the work plan and sign off on recommended
                           changes. In addition, states reported that their system users served a


                           28
                            See U.S. General Accounting Office, Executive Guide: Improving Mission Performance
                           Through Strategic Information Management and Technology, GAO/AIMD-94-115
                           (Washington, D.C.: May 1, 1994); Center for Technology in Government, University of
                           Albany, SUNY. Tying a Sensible Knot: A Practical Guide to State-Local Information
                           Systems. Albany, N.Y., June 1997.




                           Page 38                                                    GAO-03-809 Child Welfare
number of purposes, including serving as experts on the different
specialties within child welfare, such as child abuse, foster care, or
adoption, and as representatives from local or county offices to assist in
identifying the diverse approaches to capturing information across the
state. For example, Indiana reported that caseworkers involved in SACWIS
development represented the unique needs of the different geographical
areas of the state and helped design a uniform statewide system to meet
the diverse needs of large, intermediate, and small local offices.

Ten states noted that user input should not be limited to frontline workers,
such as caseworkers, but should include representatives from other areas
of the agency, such as the financial staff, and other agencies that serve
children, such as child support enforcement.29 Since many SACWIS link
with other state information systems, states advised that developing a
collaborative relationship with other state agencies will help the
development of the system. While not one of the most common challenges
reported in our survey, New Hampshire reported that one of its challenges
with meeting its SACWIS timeframe was not working collaboratively with
other agencies, such as Temporary Assistance for Needy Families
(TANF)30 and child support enforcement, to develop the payment
component of SACWIS. Similarly, we previously reported that the
difficulty developing linkages between social services agencies limits the
effectiveness of all the programs to serve families.31 To attempt to
overcome this challenge, 26 of the 46 states responding to our survey that
they are developing or operating a SACWIS indicated that they included
external public agency users and 23 reported using representatives from
other state agencies that serve children in developing their SACWIS.
Indiana said that a task force made up of representatives from the TANF
and child support enforcement agencies was developed to design the



29
  The Child Support Enforcement Program is a joint federal, state, and local partnership
that was established in 1975 under Title IV-D of the Social Security Act. Each state runs a
child support program, which provides four major services: locating non-custodial parents,
establishing paternity, establishing child support obligations, and collecting child support
for families.
30
 In 1996, the Congress created the block grant Temporary Assistance for Needy Families
program replacing the Aid to Families with Dependent Children (AFDC) and related
welfare programs. States were given increased flexibility in designing the eligibility criteria
and benefit rules, which require work in exchange for time-limited benefits.
31
 See U.S. General Accounting Office, Welfare Reform: Improving State Automated
Systems Requires Coordinated Federal Effort, GAO/HEHS-00-48 (Washington, D.C.: Apr.
27, 2000).




Page 39                                                           GAO-03-809 Child Welfare
linkages between the systems. In addition, Colorado officials reported that
they are working with the Department of Youth Corrections—an agency
that shares the SACWIS with child welfare—to ensure that the shared
screens use the same definitions.

In addition to seeking input from caseworkers and other system users
while developing SACWIS, many states continue to include users as a part
of the implementation teams, to serve as contacts in the field and provide
ongoing assistance, and to provide input on system enhancements.
Alabama responded in our survey that the state had “mentors” in each
county to help caseworkers adjust to the new system. These mentors
continue to provide ongoing support now that the system is implemented.
Similarly, Oklahoma developed Field Implementation Teams consisting of
one contractor and one child welfare staff person. During system
implementation, the teams went to field offices to provide on-site
assistance with using SACWIS and becoming accustomed with the new
method of recording child welfare information. Furthermore, Oklahoma
recruits experienced child welfare field staff for its SACWIS help desk
because of their knowledge of the system and child welfare policy and
practice.

Although states faced other challenges in completing their SACWIS, few
reported implementing approaches to overcome the barriers. According to
survey results, a common problem states faced in developing SACWIS was
receiving insufficient state funding for development. However, in our
previous work on managing information technology, we found that the IT
products can become obsolete in a matter of months rather than years,
calling for more frequent investments in upgrades and enhancements.32 In
addition, officials in Iowa told us that maintaining systems takes just as
much money as building them. States did not report in our survey
approaches for obtaining more funding for developing SACWIS, and few
states reported developing strategies in an attempt to overcome the
challenges associated with tight budgets for maintaining their systems.
For example, Iowa officials engaged in careful planning with system users
to ensure that they addressed the highest priorities when enhancing the
system. In particular, the officials reported that maintaining tight control
over the development and maintenance processes helps them avoid
investing inordinate amounts of resources to make corrections to the



32
 See U.S. General Accounting Office, Managing Technology: Best Practices Can Improve
Performance and Produce Results, GAO/T-AIMD-97-38 (Washington, D.C.: Jan. 31, 1997).




Page 40                                                    GAO-03-809 Child Welfare
                               system. In Oklahoma, child welfare officials reported that they relied on
                               the contractors who developed their SACWIS to conduct ongoing
                               maintenance activities until the contract expired in 2001. At that time, the
                               agency hired some of the contract staff as full-time state employees to
                               continue with the maintenance activities. State officials explained that this
                               approach ensured continuity of service, in addition to saving the agency
                               approximately $1 million each year. Similarly, few states reported on
                               approaches to overcome the challenge of finding contractors with
                               knowledge of child welfare practice. However, Iowa officials explained
                               that once the contract staff are hired, they are required to attend the same
                               training as new caseworkers to ensure that they are familiar with the
                               state’s child welfare policies and to familiarize themselves with casework
                               practices.


States Use Strategies, such    Twenty-eight states reported using approaches to help caseworkers
as Producing Reports That      identify the data elements that are required for federal reporting and to
Identify Missing Data, in an   help them better understand the importance of entering timely and
                               accurate data. Ten states responding to our survey reported reviewing the
Attempt to Improve the         federal reporting requirements in training sessions as a promising
Reliability of the Data        approach they use to improve data quality or as a lesson learned. For
Reported to HHS                example, Tennessee reported that the state added a component about
                               AFCARS to the initial and ongoing training workers receive about using
                               SACWIS. The curriculum addresses the AFCARS report in general and the
                               individual data elements to help the caseworkers better understand the
                               purpose of collecting the information. In Nebraska, a “desk aid” that
                               explains the data elements and where and why to enter them in the system
                               is available on the caseworkers’ computer desktops. In addition, New York
                               has developed a step-by-step guide explaining to workers how NCANDS
                               data should be entered, with references to the policy or statute requiring
                               the information.

                               To improve data reliability, some states have designed their information
                               systems with special features to encourage caseworkers to enter the
                               information. Four states responding to our survey and 3 states we visited
                               designed their SACWIS with color-coded fields to draw attention to the
                               data elements that caseworker are required to enter. For example, the
                               AFCARS data fields in Oklahoma’s system are coded red until the data are
                               entered, after which the fields change to blue. In addition, workers can
                               look at a single screen in the Oklahoma system to see what AFCARS data
                               elements need to be completed without having to scroll through the entire
                               case record. Colorado, Iowa, New York, and Oklahoma have built into
                               their systems alerts—also known as “ticklers”—to remind caseworkers


                               Page 41                                              GAO-03-809 Child Welfare
             and supervisors of tasks that they need to complete. For example, in Iowa,
             alerts are sent to supervisors if a caseworker fails to enter the data
             necessary to complete a payment to a foster care provider. Whereas, in
             Oklahoma, a stoplight icon on the caseworker’s computer desktop
             reminds the worker when tasks are due. A green light indicates that
             nothing is due within 5 days; a yellow light means that something is due
             within 5 days; and a red light means that something is overdue.
             Caseworkers and supervisors in the states we visited had mixed responses
             about the usefulness and effectiveness of the alerts. Some caseworkers
             found them to be a nuisance, while other caseworkers and supervisors
             found them to be useful tools in managing workloads and prioritizing daily
             tasks.

             Six states reported that the best way to improve data quality was to use
             the data in published reports and hold the caseworkers and supervisors
             accountable for the outcomes of the children in their care. In addition, 6
             states responding to our survey reported using the data available in their
             information systems to measure state outcomes similar to the CFSR. State
             officials reported that this approach is an effective way to get local offices
             invested in the quality of the data. For example, North Carolina publishes
             monthly reports for each county comparing their performance on state
             data indicators, such as the length of time children spend in care, to
             counties of similar size and the state as a whole. County officials reported
             that these reports encourage workers to improve the quality of the data
             collected and entered into the state system since their performance is
             being widely published and compared to other counties.

             In addition, all the states we visited reported that frequent review of their
             data, such as using software from HHS to test their AFCARS and NCANDS
             data to pin-point data entry errors prior to submitting them to HHS, has
             helped improve data quality. When the states identify poor data, they alert
             the caseworkers and supervisors of needed corrections and data entry
             improvements. For example, Colorado runs these reports about 4 to 5
             times a year, with one run occurring approximately 6 weeks before each
             AFCARS submission. When the data specialists find errors, they notify the
             caseworker to clean up the data. New York officials told us that they
             incorporate the results from these tests in training if a consistent pattern
             of errors is identified.


             While most states are developing statewide information systems,
Conclusion   challenges with data reliability remain. Although SACWIS development is
             delayed in many states, state officials recognize the benefits of having a


             Page 42                                               GAO-03-809 Child Welfare
                    uniform system that enhances the states’ ability to monitor the services
                    provided and the outcomes for children in their care. Although states
                    began reporting to NCANDS in1990 and were mandated to begin reporting
                    to AFCARS in 1995, most states continue to face challenges providing
                    complete, accurate, and consistent data to HHS. In addition, the results of
                    more recent HHS efforts, such as conducting AFCARS-related focus
                    groups, are unknown. Reliable data are essential to the federal
                    government’s development of policies that address the needs of the
                    children served by state child welfare agencies and its ability to assist
                    states in improving child welfare system deficiencies. Without well-
                    documented, clearer guidance and the completion of more comprehensive
                    reviews of states’ AFCARS reporting capabilities, states are limited in
                    overcoming challenges that affect data reliability. Because these
                    challenges still remain, HHS may be using some questionable data as the
                    foundation for national reports and national standards for the CFSR and
                    may not have a clear picture of how states meet the needs of children in
                    their care.


                    To improve the reliability of state-reported child welfare data, we are
Recommendation to   recommending that the Secretary of HHS consider, in addition to HHS’s
the Secretary of    recent efforts to improve AFCARS data, ways to enhance the guidance and
                    assistance offered to states to help them overcome the key challenges in
Health and Human    collecting and reporting child welfare data. These efforts could include a
Services            stronger emphasis placed on conducting AFCARS reviews and more
                    timely follow-up to help states implement their improvement plans or
                    identifying a useful method to provide clear and consistent guidance on
                    AFCARS and NCANDS reporting.


                    We obtained comments on a draft of this report from the Department of
Agency Comments     Health and Human Services’ Administration for Children and Families
                    (ACF). These comments are reproduced in appendix III. ACF also
                    provided technical clarifications, which we incorporated when
                    appropriate.

                    ACF generally agreed with our findings and commented that the report
                    provides a useful perspective of the problems states face in collecting data
                    and of ACF’s effort to provide ongoing technical assistance to improve the
                    quality of child welfare data. In response to our recommendation, ACF
                    said that we categorized its efforts as “recent” and did not recognize the
                    long-term efforts to provide AFCARS and NCANDS related guidance to the
                    states. Although we did not discuss each effort in depth, we do mention


                    Page 43                                              GAO-03-809 Child Welfare
the agency’s ongoing efforts in our report. However, we refer to the recent
efforts in the recommendation in recognition of the agency’s current
activities to formally obtain, document, and incorporate feedback from the
states with regard to collecting and reporting adoption and foster care
data. ACF also noted in its comments that the data definitions need to be
updated and revised and said it is currently in the process of revising the
AFCARS regulations to further standardize the information states are to
report—which we acknowledge in our report. In addition to the steps HHS
is taking to further improve the AFCARS data, our recommendation
encourages HHS to consider ways to enhance the ongoing guidance and
assistance offered to states to help them overcome the key challenges in
collecting and reporting child welfare data. ACF requested specific
recommendations on approaches to overcome the difficulty of collecting
and merging information from multiple state and county programs into a
single national database. While there may be additional methodologies
that the agency could use to overcome such challenges, our
recommendation focuses on improving the guidance already offered to the
states as a step to helping them better comply with the reporting
requirements.

In addition, ACF added that although staff turnover in state child welfare
agencies is a significant contributor to data quality issues, we did not focus
on this as a significant factor. ACF also commented that it is firmly
committed to continue to support the states and to provide technical
assistance and other guidance as its resources will permit. However,
because we recently issued a detailed report on a variety of caseworker
issues,33 we primarily focused in this report on the key data entry
challenges caseworkers face and refer readers to our previous work for
additional information on challenges related to caseworker recruitment
and retention and their affect on child welfare agencies. In commenting on
our previous work prior to its release, HHS indicated that it does not have
the authority to require states to address factors that contribute to staff
turnover, such as high caseloads and said that it has limited resources to
assist the states in the area of staff recruitment and retention.

ACF commented that it provided increased funding to the National
Resource Centers in fiscal year 2003, which they believe will improve



33
 See U.S. General Accounting Office, Child Welfare: HHS Could Play a Greater Role in
Helping Child Welfare Agencies Recruit and Retain Staff. GAO-03-357. (Washington, D.C.:
Mar. 31, 2003).




Page 44                                                      GAO-03-809 Child Welfare
ACF’s ability to provide assistance to the states. After receiving the draft
report for comment, HHS separately provided information on an
additional service the National Resource Center for Information
Technology in Child Welfare provides to states. In an effort to assist states
with improving the quality of their AFCARS data, the Resource Center will
review states’ programming code used for AFCARS data. As of June 2003,
HHS reported that the Resource Center provided this assistance to
Arkansas, Louisiana, Mississippi, North Carolina, Nevada, New Jersey, and
Rhode Island, and 3 states—Maryland, Michigan, and Wisconsin—and the
District of Columbia have requested the assistance.

In response to our survey methodology, ACF requested that we explain
why the territory of Puerto Rico was not included in the state survey.
Although Puerto Rico receives federal child welfare funds, we traditionally
focus on the states and therefore do not include the U.S. territories,
including American Samoa, the Commonwealth of the Northern Mariana
Islands, Guam, Puerto Rico, and the Virgin Islands, in the scope of our
reviews.

Finally, in response to our discussion of the AFCARS review process, ACF
provided a few clarifications. During the course of our review, an HHS
official characterized the AFCARS review process as relatively new and
explained that the agency is still developing a process to respond to the
states following the completion of the on-site review. When responding to
a draft of this report, ACF disagreed with this characterization. ACF
commented that the review process has been in place since 1996, pointing
to the pilot reviews as evidence that the agency has a defined process.
However, when we requested AFCARS reports for review, HHS explained
that the states undergoing pilot reviews would be re-reviewed and that the
official process was formalized in 2001 with the release of an AFCARS
review guide and the start of the official reviews. In addition, ACF
commented that SACWIS reviews do not take priority over AFCARS
reviews. However, officials had previously explained that although
SACWIS and AFCARS reviews can happen at the same time, in practice,
the AFCARS reviews are scheduled to occur in the states that are
developing SACWIS after they have participated in a SACWIS review.
Furthermore, ACF explained that states do not develop their improvement
plan following the conclusion of the AFCARS review. Instead, ACF
officials draft the plan for the state. Although state representatives had
described a challenge in receiving timely feedback on their improvement
plan, we have changed the language in the report to reflect ACF’s
comment.



Page 45                                              GAO-03-809 Child Welfare
We also provided a copy of our draft to child welfare officials in the 5
states we visited—Colorado, Iowa, North Carolina, New York, and
Oklahoma. Iowa and New York had technical clarifications, which we in
incorporated when appropriate. Oklahoma provided additional
information, which was incorporated. Colorado had no suggested
corrections or edits. North Carolina did not provide any comments.


As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days after its
issue date. At that time, we will send copies of this report to the Secretary
of Health and Human Services, state child welfare directors, and other
interested parties. We will make copies available to others on request. In
addition, the report will be available at no charge on GAO’s Web site at
http://www.gao.gov. If you or your staff have any questions or wish to
discuss this material further, please call me at (202) 512-8403 or Diana
Pietrowiak at (202) 512-6239. Key contributors to this report are listed in
appendix IV.




Cornelia M. Ashby
Director, Education, Workforce,
 and Income Security Issues




Page 46                                                GAO-03-809 Child Welfare
             Appendix I: Scope and Methodology
Appendix I: Scope and Methodology


             To determine the progress states have made in developing Statewide
             Automated Child Welfare Information Systems (SACWIS), we surveyed all
             50 states and the District of Columbia through a Web-based survey. We
             pretested the survey instrument in Maryland and the District of Columbia.
             We received responses from 49 states and the District of Columbia. The
             state of Nevada did not respond to the survey. We discarded a question
             that asked states to report the date their advance planning document
             (APD) was approved by the Department of Health and Human Services
             (HHS). Due to a technical error, the date was truncated and a valid answer
             was not stored in the responses. Of the 50 survey responses, 46 were from
             states that are pursuing SACWIS development. The 4 states not developing
             SACWIS were asked to skip sections of the survey that asked about
             SACWIS development, system modifications, and supported services and
             links. We did not independently verify the survey responses. In addition,
             we visited 5 states to obtain more detailed and qualitative information
             regarding states’ experiences developing SACWIS. We conducted site
             visits in Colorado, Iowa, New York, North Carolina, and Oklahoma. We
             selected these states to represent a range of SACWIS development stages,
             sizes of foster care populations, and geographic locations. During our site
             visits, we interviewed state and local child welfare staff, state and local
             staff that regularly exchange information with the child welfare agency,
             and private contractors. We also spoke with HHS staff in the central and
             regional offices, National Resource Center officials, contractors involved
             in SACWIS development, and child welfare experts from the Child Welfare
             League of America and the American Public Human Services Association.

             To determine how states and HHS ensure reliable data exist on children
             served by child welfare agencies we surveyed states using the above-
             mentioned survey instrument. In addition, we interviewed state and HHS
             officials on their efforts to analyze and compile data and HHS’s role in
             providing technical assistance to states. We spoke with state officials
             during our site visits and HHS officials in the central and regional offices
             and attended the 6th National Child Welfare Data Conference. We
             obtained and reviewed available SACWIS and Adoption and Foster Care
             Analysis and Reporting System (AFCARS) reports. At the time of our
             review, HHS had conducted 26 SACWIS reviews. We obtained and
             reviewed 23 reports. The remaining reports were not available for review
             because HHS has not yet completed the report or shared the results with
             the state. Most of the SACWIS reports were considered drafts since many
             states are in the process of resolving issues with completing their systems.
             We reviewed AFCARS assessment reports from 6 of the 8 states assessed
             by HHS—Arkansas, Connecticut, New Mexico, Texas, Vermont, and
             Wyoming. HHS conducted reviews in Delaware and West Virginia after we


             Page 47                                              GAO-03-809 Child Welfare
Appendix I: Scope and Methodology




completed our analysis. We did not review any of the eight pilot review
reports since these were not final reports and HHS plans to conduct
official reviews in these states. These AFCARS assessment reports were
analyzed to understand the breadth of on-site assistance HHS provides to
states during the review and to identify common data collection and
reporting difficulties among states. Finally, we talked with officials in 6 of
the 8 states that had an AFCARS review about their experiences during the
review and child welfare experts.

To identify practices state and local child welfare agencies are using to
help ensure the accuracy, timeliness, and completeness of child welfare
data we interviewed state and local child welfare officials on our site visits
and inquired about the practices they are employing. We also included
questions on practices and lessons learned in our survey. In addition, we
spoke with numerous child welfare experts, including individuals from the
National Resource Center for Information Technology in Child Welfare,
the Child Welfare League of America, and the American Public Human
Services Association.1




1
 The Child Welfare League of America is an association of almost 1,200 public and private
nonprofit agencies that assist abused and neglected children and their families with a wide
range of services. The American Public Human Services Association is an organization of
individuals and agencies concerned with human services, including state and local human
service agencies and individuals who work in or otherwise have an interest in human
service programs.




Page 48                                                         GAO-03-809 Child Welfare
              Appendix II: State SACWIS Stages of
Appendix II: State SACWIS Stages of
              Development



Development


               SACWIS stages of development
               Completed                            Partially operational
               Arizona                              Alabama
               Massachusetts                        Florida
               Oklahoma                             Illinois
               Rhode Island                         Mississippi
               West Virginia                        New York
                                                    Oregon
               Operational                          South Carolina
               Arkansas                             Tennessee
               California                           Wisconsin
               Colorado
               Connecticut                          Implementation
               Delaware                             Alaska
               District of Columbia                 Maryland
               Idaho
               Indiana                              Planning
               Iowa                                 Georgia
               Kentucky                             Kansas
               Maine                                Louisiana
               Michigan                             Missouri
               Minnesota                            New Jersey
               Montana                              Ohio
               Nebraska                             Pennsylvania
               Nevada
               New Hampshire                        No SACWIS
               New Mexico                           Hawaii
               South Dakota                         North Carolina
               Texas                                North Dakota
               Utah                                 Vermont
               Virginia
               Washington
               Wyoming
              Source: HHS.

              Note: Status is as of May 22, 2003.




              Page 49                                          GAO-03-809 Child Welfare
             Appendix III: Comments from the Department of Health and Human Services
Appendix III: Comments from the
Department of Health and Human Services




             Page 50                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 51                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 52                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 53                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 54                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 55                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 56                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 57                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 58                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 59                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 60                                                    GAO-03-809 Child Welfare
Appendix III: Comments from the Department of Health and Human Services




Page 61                                                    GAO-03-809 Child Welfare
                  Appendix IV: GAO Contacts and
Appendix IV: GAO Contacts and
                  Acknowledgments



Acknowledgments

                  Diana Pietrowiak, (202) 512-6239
GAO Contacts      Sara L. Schibanoff, (202) 512-4176


                  In addition to those named above, Leah DeWolf and Rachel Seid made key
Staff             contributions to this report. Avrum Ashery, Patrick DiBattista,
Acknowledgments   Barbara Johnson, Valerie Melvin, and Rebecca Shea also provided key
                  technical assistance.




                  Page 62                                          GAO-03-809 Child Welfare
               Bibliography
Bibliography


               The American Public Welfare Association. Statewide Automated Child
               Welfare Information Systems: Survey of State Progress. Washington,
               D.C., July 1997.

               The American Public Welfare Association. Child Welfare Information
               Systems: Some Concepts and Their Implications. Washington, D.C., July
               1994.

               The American Public Welfare Association. Survey of State Child Welfare
               Information Systems: Status of AFCARS and SACWIS. Washington, D.C.,
               April 1995.

               Caliber Associates, Analysis of State Child Welfare Data: VCIS Survey
               Data from 1990 through 1994, May 1998, Department of Health and
               Human Services.

               Center for Technology in Government, University of Albany, SUNY. Tying
               a Sensible Knot: A Practical Guide to State-Local Information Systems.
               Albany, N.Y., June 1997.

               Child Welfare League of America. National Working Group Highlights,
               “Child Maltreatment in Foster Care: Understanding the Data.” Washington,
               D.C., October 2002.

               Child Welfare League of America. National Working Group Highlights,
               “Placement Stability Measure and Diverse Out-of-Home Care Populations.”
               Washington, D.C., April 2002.

               U.S. Department of Health and Human Services, Administration for
               Children and Families, Administration on Children, Youth and Families,
               Children’s Bureau. Child Maltreatment 2001. Washington, D.C., 2003.

               U.S. Department of Health and Human Services, Administration for
               Children and Families, Administration on Children, Youth and Families,
               Children’s Bureau. Child Welfare Outcomes 1999: Annual Report.
               Washington, D.C., n.d..

               U.S. Department of Health and Human Services, Office of Inspector
               General. Adoption and Foster Care Analysis and Reporting System
               (AFCARS): Challenges and Limitations. Washington, D.C., March 2003.




               Page 63                                            GAO-03-809 Child Welfare
             Related GAO Products
Related GAO Products


             Child Welfare and Juvenile Justice: Federal Agencies Could Play a
             Stronger Role in Helping States Reduce the Number of Children Placed
             Solely to Obtain Mental Health Services. GAO-03-397. Washington, D.C.:
             April 21, 2003.

             Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare
             Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March
             31, 2003.

             Human Services: Federal Approval and Funding Processes for States’
             Information Systems. GAO-02-347T. Washington, D.C.: July 9, 2002.

             Foster Care: Recent Legislation Helps States Focus on Finding
             Permanent Homes for Children, but Long-Standing Barriers Remain.
             GAO-02-585. Washington, D.C.: June 28, 2002.

             Human Services Integration: Results of a GAO Cosponsored Conference
             on Modernizing Information Systems. GAO-02-121. Washington, D.C.:
             January 31, 2002.

             District of Columbia Child Welfare: Long-Term Challenges to Ensuring
             Children’s Well-Being. GAO-01-191. Washington, D.C.: December 29, 2000.

             Child Welfare: New Financing and Service Strategies Hold Promise, but
             Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.

             Welfare Reform: Improving State Automated Systems Requires
             Coordinated Federal Effort. GAO/HEHS-00-48. Washington, D.C.: April 27,
             2000.

             Foster Care: States’ Early Experiences Implementing the Adoption and
             Safe Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.

             Foster Care: HHS Could Better Facilitate the Interjurisdictional
             Adoption Process. GAO/HEHS-00-12. Washington, D.C.: November 19,
             1999.

             Foster Care: Effectiveness of Independent Living Services Unknown.
             GAO/HEHS-00-13. Washington, D.C.: November 5, 1999.

             Foster Care: Kinship Care Quality and Permanency Issues. GAO/HEHS-
             99-32. Washington, D.C.: May 6, 1999.



             Page 64                                           GAO-03-809 Child Welfare
           Related GAO Products




           Juvenile Courts: Reforms Aim to Better Serve Maltreated Children.
           GAO/HEHS-99-13. Washington, D.C.: January 11, 1999.

           Child Welfare: Early Experiences Implementing a Managed Care
           Approach. GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.

           Foster Care: Agencies Face Challenges Securing Stable Homes for
           Children of Substance Abusers. GAO/ HEHS-98-182. Washington, D.C.:
           September 30, 1998.

           Managing Technology: Best Practices Can Improve Performance and
           Produce Results. GAO/T-AIMD-97-38, January 31, 1997.

           Child Welfare: HHS Begins to Assume Leadership to Implement National
           and State Systems. GAO/AIMD-94-37. Washington, D.C.: June 8, 1994.

           Executive Guide: Improving Mission Performance Through Strategic
           Information Management and Technology. GAO/AIMD-94-115. May 1,
           1994.




(130163)
           Page 65                                          GAO-03-809 Child Welfare
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily
                         E-mail alert for newly released products” under the GAO Reports heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548