oversight

Child Welfare: States Face Challenges in Developing Information Systems and Reporting Reliable Child Welfare Data

Published by the Government Accountability Office on 2003-11-19.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                               United States General Accounting Office

GAO                            Testimony
                               Before the Subcommittee on Human
                               Resources, Committee on Ways and
                               Means, House of Representatives

For Release on Delivery
Expected at 2:00 p.m. EST
Wednesday, November 19, 2003   CHILD WELFARE
                               States Face Challenges in
                               Developing Information
                               Systems and Reporting
                               Reliable Child Welfare Data
                               Statement of Cornelia M. Ashby, Director,
                               Education, Workforce, and Income Security Issues




GAO-04-267T
                                                November 19, 2003


                                                CHILD WELFARE

                                                States Face Challenges in Developing
Highlights of GAO-04-267T, a testimony          Information Systems and Reporting
for the Subcommittee on Human
Resources, Committee on Ways and                Reliable Child Welfare Data
Means, House of Representatives




To better monitor children and                  HHS reported that 47 states are developing or operating a SACWIS, but many
families served by state child                  states continue to face challenges developing their systems. Most state
welfare agencies, Congress                      officials said they recognize the benefit their state will achieve by developing
authorized matching funds for the               SACWIS, such as contributing to the timeliness of child abuse and neglect
development of statewide                        investigations; however, despite the availability of federal funds since 1994,
automated child welfare
information systems (SACWIS) and
                                                states reported a median delay of 2½ years beyond the time frames they set
required that the Department of                 for completion. States reported that they encountered some difficulties
Health and Human Services (HHS)                 during SACWIS development, such as challenges receiving state funding and
compile information on the                      creating a system that reflected their work processes. In response to some
children served by state agencies.              of these challenges, HHS has provided technical assistance to help states
This testimony is based on our July             develop their systems and conducted on-site reviews of SACWIS to verify
2003 report and addresses the                   that the systems meet federal requirements.
following: (1) states’ experiences in
developing child welfare                        Despite efforts to implement comprehensive information systems, several
information systems and HHS’s                   factors affect the states’ ability to collect and report reliable adoption, foster
role in assisting in their                      care, and child abuse and neglect data. States responding to GAO’s survey
development, (2) factors that affect
the reliability of data that states
                                                and officials in the five states GAO visited reported that insufficient
collect and report on children                  caseworker training and inaccurate and incomplete data entry affect the
served by their child welfare                   quality of the data reported to HHS. In addition, states reported technical
agencies and HHS’s role in                      challenges reporting data. Despite HHS assistance, many states report
ensuring the reliability of those               ongoing challenges, such as the lack of clear and documented guidance on
data, and (3) practices that child              how to report child welfare data. In addition, although states were
welfare agencies use to overcome                mandated to begin reporting data to the Adoption and Foster Care Analysis
challenges associated with SACWIS               Reporting System (AFCARS) in 1995, few reviews of states’ AFCARS
development and data reliability.               reporting capabilities have been conducted to assist states in resolving some
For the July 2003 report, we                    of their reporting challenges.
surveyed all 50 states and the
District of Columbia regarding their
experiences developing and using
                                                Some states are using a variety of practices to address the challenges
information systems and their                   associated with developing SACWIS and improving data reliability. For
ability to report data to HHS. We               example, 28 states reported using approaches to help caseworkers identify
also reviewed a variety of HHS                  and better understand the data elements that are required for federal
documents and visited five states               reporting.
to obtain firsthand information.
Finally, we interviewed HHS                     In a related report, we recommended that the Secretary of HHS consider
officials and child welfare and data            ways to enhance the guidance and assistance offered to states to help them
experts and reviewed relevant                   overcome the key challenges in collecting and reporting child welfare data.
literature.                                     These efforts could include a stronger emphasis placed on conducting
                                                AFCARS reviews and timelier follow-up to help states implement their
                                                improvement plans or identifying a useful method to provide clear and
                                                consistent guidance. HHS generally agreed with our findings but, in
                                                response to our recommendation, said that we did not recognize the long-
                                                term efforts to provide AFCARS and National Child Abuse and Neglect Data
www.gao.gov/cgi-bin/getrpt?GAO-04-267T.
                                                System related guidance. HHS also noted that the data definitions need to be
To view the full product, including the scope   updated and revised and said it was in the process of revising regulations.
and methodology, click on the link above.       HHS added that it is firmly committed to continue to support the states and
For more information, contact Cornelia M.
Ashby at (202) 512-8403 or
                                                to provide technical guidance and assistance as resources permit.
ashbyc@gao.gov.
Mr. Chairman and Members of the Subcommittee:

Thank you for inviting me here today to discuss states’ development of
automated child welfare information systems. As you are aware, the
Congress required that the Department of Health and Human Services
(HHS) compile information on the children served by state agencies and
authorized federal funds to match those of states for use in the
development of state child welfare information systems. Since 1994,
designated federal matching funds have been available to states to develop
and implement comprehensive case management systems—statewide
automated child welfare information systems (SACWIS)—to manage their
child welfare cases as well as to report child abuse and neglect, foster
care, and adoption information to the federal government. States have the
option to implement a SACWIS or develop different information systems
without using SACWIS funds to support their child welfare agencies and
collect information on their child welfare cases. Regardless of the type of
system a state develops, child welfare caseworkers at the county or local
level are the key personnel who collect and document information on
children and families served by child welfare agencies, in addition to
performing a wide range of services to protect children—such as
investigating child abuse or neglect reports or providing support services
to maintain the children in their homes.

Currently, HHS compiles state-reported child welfare data in two
databases: the Adoption and Foster Care Analysis and Reporting System
(AFCARS) and the National Child Abuse and Neglect Data System
(NCANDS). HHS relies on the information available in its databases to
analyze and track children’s experiences in the child welfare system, to
determine states’ performance on federal child welfare outcome measures,
and to report to Congress on children’s well being and child welfare
experiences.

My testimony today will focus on three key issues: (1) states’ experiences
in developing child welfare information systems and HHS’s role in
assisting in their development; (2) factors that affect the reliability of data
that states collect and report on children served by their child welfare
agencies, and HHS’s role in ensuring the reliability of those data; and (3)
practices that child welfare agencies use to overcome challenges
associated with SACWIS development and data reliability. My comments
are based on the findings from our July 2003 report, Child Welfare: Most
States Are Developing Statewide Information Systems, but the
Reliability of Child Welfare Data Could Be Improved (GAO-03-809, July
31, 2003). Those findings were based on our survey of all 50 states and the

Page 1                                                             GAO-04-267T
District of Columbia regarding their experiences in developing and using
information systems and their ability to report data to HHS. We received
responses from 49 states and the District of Columbia,1 although some
states did not respond to every question. We also reviewed a variety of
HHS documents, including the protocol and reports for its reviews of
SACWIS systems and states’ AFCARS reporting capabilities and visited
five states—Colorado, Iowa, New York, North Carolina, and Oklahoma—
to obtain firsthand information on their experiences developing SACWIS
and reporting data to HHS. We selected these states to represent
geographic diversity and different stages of SACWIS implementation.
Finally, we interviewed HHS officials and child welfare and data experts
and reviewed relevant literature.

In summary, HHS reported that 47 states were developing or operating a
SACWIS, but many states continue to face challenges developing their
systems. Most state officials said they recognize the benefit their state will
achieve by developing SACWIS, but added that they have encountered
difficulties in receiving state funding and in creating a system that
reflected their work processes. Despite the availability of federal funds
since 1994, states reported a median delay of 2½ years beyond the time
frames they set for completion. Several factors affect the states’ ability to
collect and report reliable adoption, foster care, and child abuse and
neglect data. For example, insufficient caseworker training and inaccurate
and incomplete data entry affect the quality of data reported to HHS.
States also reported technical challenges reporting data. Despite HHS’s
assistance, many states reported ongoing challenges, such as the lack of
clear and documented guidance from HHS on how to report child welfare
data. In addition, although states were mandated to begin reporting data to
AFCARS in 1995, few reviews of states’ AFCARS reporting capabilities
have been conducted. Some states are using a variety of practices to
address the challenges they face in developing SACWIS and improving
data reliability. For example, 28 states reported using approaches to help
caseworkers identify and better understand the data elements that are
required for federal reporting. To improve the reliability of state-reported
child welfare data, we recommended in our July 2003 report that the
Secretary of HHS consider ways to enhance the guidance and assistance



1
 Throughout this testimony, references to state survey responses include the District of
Columbia. Forty-six of these states reported that they are developing or operating a
SACWIS. Nevada, which HHS reported has an operational SACWIS, did not respond to our
survey.



Page 2                                                                     GAO-04-267T
                 offered to states to help them overcome the key challenges in collecting
                 and reporting child welfare data.


                 ACF’s Children’s Bureau is responsible for the administration and
Background       oversight of federal funding to states for child welfare services under
                 Titles IV-B and IV-E of the Social Security Act. However, the monitoring of
                 children served by state child welfare agencies is the responsibility of the
                 state agencies that provide the services to these children and their
                 families. Child welfare caseworkers at the county or local level are the key
                 personnel responsible for documenting the wide range of services offered
                 to children and families, such as investigations of abuse and neglect,
                 treatment services offered to keep families intact and prevent the need for
                 foster care, and arrangements made for permanent or adoptive placements
                 when children must be removed from their homes. Caseworkers are
                 supported by supervisors, who typically assign new cases to workers and
                 monitor caseworkers’ progress in achieving desired outcomes, analyzing
                 and addressing problems and making decisions about cases.

                 To qualify for federal funding for SACWIS, states must prepare and submit
                 an advance planning document (APD) to the Children’s Bureau, in which
                 they describe the state’s plan for managing the design, development,
                 implementation, and operation of a SACWIS that meets federal
                 requirements and state needs in an efficient, comprehensive, and cost-
                 effective manner. In addition, the state must establish SACWIS and
                 program performance goals in terms of projected costs and benefits in the
                 APD. States are required to submit separate APDs for the planning and
                 development phases, in addition to periodic updates.

                 Since the administration and structure of state child welfare agencies vary
                 across the nation, states can design their SACWIS to meet their state
                 needs, as long as states meet certain federal requirements. Federal funding
                 is available to states for SACWIS that

             •   meet the requirements for reporting AFCARS data to HHS;

             •   to the extent practicable, are capable of linking with the state data
                 collection system that collects information on child abuse and neglect;

             •   to the extent practicable, are capable of linking with, and retrieving
                 information from, the state data collection system that collects
                 information on the eligibility of individuals under Title IV-A—Temporary
                 Assistance for Needy Families; and


                 Page 3                                                          GAO-04-267T
•   provides for more efficient, economical, and effective administration of
    the programs carried out under a state’s plans approved under Titles IV-B
    and IV-E of the Social Security Act.

    A SACWIS must operate uniformly as a single system in each state and
    must encompass all entities that administer programs provided under
    Titles IV-B and IV-E. In some cases, HHS will allow the statewide system
    to link to another state system to perform required functions, such as
    linking to financial systems to issue and reconcile payments to child
    welfare service providers. The state’s APD must describe how its SACWIS
    will link to other systems to meet the requirements in the SACWIS
    regulations.

    In addition to monitoring the APDs of the states that are developing
    SACWIS, HHS reviews state information systems through formal SACWIS
    assessment reviews and the Child and Family Services Reviews (CFSR)—a
    federal review process to monitor states’ compliance with child welfare
    laws and federal outcome measures. The formal SACWIS reviews are
    conducted by the Children’s Bureau to determine if a state has developed
    and implemented all components detailed in the state’s APD and if the
    system adheres to federal requirements. The CFSR assesses statewide
    information systems, along with other systemic factors, to determine if the
    state is operating a system that can readily identify the status,
    demographic characteristics, location, and goals for placement of every
    child who is in foster care. This systemic factor is reviewed in all states,
    regardless of whether the state is developing a SACWIS or the stage of
    system development. For the 40 CFSR reports that are available, HHS
    found that four states were not in substantial conformity on the statewide
    information system indicator.2 These four states must address how they
    will come into conformity with this factor in a program improvement plan.
    HHS has also conducted SACWIS reviews in two of these states.




    2
     We are currently conducting an engagement on states’ and HHS’s experiences in
    conducting the CFSRs.



    Page 4                                                                    GAO-04-267T
                           While 47 states are developing or operating a SACWIS, many challenges
Most States Are            remain despite HHS’s oversight and technical assistance. Since 1994,
Developing SACWIS,         states reported that they have spent approximately $2.4 billion in federal,
                           state, and local funding on SACWIS. While most state officials we
But Challenges             interviewed and those responding to our survey said that they recognize
Remain Despite HHS’s       the benefits their state will achieve by developing a statewide system,
                           many states reported that the development of their SACWIS is delayed
Oversight and              between 2 months and 8 years beyond the time frames the states set for
Technical Assistance       completion, with a median delay of 2½ years. Most states responding to
                           our survey faced challenges, such as obtaining state funding and
                           developing a system that met the child welfare agency’s needs statewide.
                           In response to some of these challenges, HHS has provided technical
                           assistance to help states develop their systems and conducted on-site
                           SACWIS reviews to verify that the systems meet all federal requirements.


States Are Using Federal   Currently, 47 states are developing or operating a SACWIS and are in
and State Funds and        various stages of development—ranging from planning to complete. The
Various Participants to    states responding to our survey reported using approximately $1.3 billion
                           in federal funds3 and approximately $1.1 billion in state and local funds4 for
Develop Multicomponent     their SACWIS. However, HHS estimated that it allocated approximately
SACWIS                     $821 million between fiscal years 1994 and 2001 in SACWIS developmental
                           funds5 and $173 million between fiscal years 1999 and 2001 in SACWIS
                           operational funds.6 The total amount of federal funding provided to states
                           for SACWIS is unknown because states claimed operational costs as a part



                           3
                            Forty-four states provided information on the total amount of federal funds they received
                           to develop and operate SACWIS. Alaska, Hawaii, Missouri, North Carolina, Texas, and
                           Vermont did not report federal funding information. Nevada did not respond to our survey.
                           State-reported figures may include some funding allocated in fiscal year 2003, since the
                           survey was issued in October 2002 and completed as late as December 2002.
                           4
                            Forty-four states provided information on the total amount of state funds used to develop
                           and operate SACWIS. Arkansas, Hawaii, Missouri, North Carolina, Texas, and Vermont did
                           not report state funding information. Nevada did not respond to our survey. State-reported
                           figures may include some funding allocated in fiscal year 2003 since the survey was issued
                           in October 2002 and completed as late as December 2002.
                           5
                           This figure includes developmental funds allocated by HHS to 49 states and the District of
                           Columbia. Hawaii did not take any federal money for SACWIS development.
                           6
                            This figure includes operational funds allocated to 35 states. States begin claiming
                           operational costs when some or all components of their SACWIS are operating in local
                           offices. Operational activities include routine maintenance, minor enhancements, and other
                           changes that do not significantly increase or modify the functionality of the system.



                           Page 5                                                                      GAO-04-267T
of their Title IV-E administrative expenses prior to 1999.7 Although the
federal government matched state funding at an enhanced rate of
75 percent beginning in 1994, many states did not apply for federal funding
or begin SACWIS development until 1996 or 1997, when more than
$467 million—the bulk of federal funds—were allocated. Most states were
still developing their SACWIS by the time enhanced funding expired in
1997, after which states could receive a 50 percent federal financial
participation for SACWIS development and operation. Although 47 states
are currently developing or operating a SACWIS, all states except Hawaii
received some federal SACWIS funds. For example, according to figures
provided by HHS, North Carolina and North Dakota received some
developmental funds but encountered difficulties that prevented them
from completing their systems.

In order to track states’ SACWIS development, HHS places them in six
categories that identify their stage of development (see table 1). HHS
sometimes recategorizes states into a lower stage of development when
problems are encountered. In addition, while HHS may classify a state
system as complete following an assessment of the state’s SACWIS, a state
may make additional changes to the system since SACWIS, like other
computer systems, continually evolve as technology and child welfare
practices change. States can claim federal funding for these changes as
operational expenses. An HHS official reported that such changes do not
need prior approval unless they are in excess of $5 million.




7
 According to HHS officials, prior to fiscal year 2000, states reported SACWIS operational
expenses as part of their Title IV-E administrative expenses because the claims sheet states
used for reporting did not have a separate column for SACWIS operational expenditures. In
fiscal year 2000, states were required to use a claims sheet that was reformatted to provide
space for SACWIS operational expenditures. In addition, an HHS official explained that the
difference between the state-reported figures and the federal figures may be due to states
claiming some SACWIS expenses under different programs, such as Title IV-E
administrative funds, rather than separately as SACWIS expenses.



Page 6                                                                       GAO-04-267T
Table 1: Number of States in Various Stages of SACWIS Development

    Stage                                                                              Number of states
                   a
    Complete                                                                                           5
                       b
    Operational                                                                                        24
                                   c
    Partially operational                                                                              9
                               d
    Implementation                                                                                     2
               e
    Planning                                                                                           7
                           f
    No SACWIS                                                                                          4
Source: HHS.

Note: Status is as of October 13, 2003.
a
 The SACWIS assessment process is completed, and all functional requirements and specifications
set forth in the APD are either included in the system or in an accepted corrective action plan.
b
 All functional requirements and specifications in the APD are included in the system, and the system
is functional statewide, but state has not completed a SACWIS assessment or is working on other
issues.
c
The state is still rolling out a system to field sites or still adding functions to systems that are
operational statewide.
d
    In active design and development, even if delayed while waiting to resolve problems such as funding.
e
    Working through options for a SACWIS.
f
    Have never pursued SACWIS funding or have abandoned plans to develop a system.



States have considerable flexibility in the design of their SACWIS.
According to HHS officials, a state should be using its SACWIS as a case
management tool that uses automation to support the various aspects of
state child welfare programs, such as recording child protection, out-of-
home care, and foster care and adoption services. To further assist child
welfare practice, states have designed their systems to follow the natural
flow of child welfare practice in their state and have added design features
to help track key events during a case. For example, in Iowa child welfare
work is divided between child abuse and neglect investigations and
ongoing case management for children brought into the care of the child
welfare agency. As a result, Iowa designed a SACWIS to reflect this work
process by linking two databases—one to record child abuse and neglect




Page 7                                                                                       GAO-04-267T
information and one to record ongoing case records—that share
information with each other.8

Since many states are in different phases of SACWIS development, their
systems currently support to varying degrees a variety of child welfare and
administrative components (see table 2). According to HHS, while the
components listed in table 2 are required for a state’s SACWIS to be
considered compliant with federal guidance—either through an interface
or built within the system—some of the subcomponents, such as a
function that helps caseworkers manage their caseloads, are optional.
HHS has encouraged states to automate as many functions as possible in
the SACWIS in an effort to cut down on the additional paperwork or
duplicative steps inherent in manual data collection.




8
 Although the Iowa state officials described their SACWIS as including the child abuse and
neglect system, HHS commented on a draft of the July 2003 report that it does not view the
child abuse and neglect system as part of the state’s SACWIS. However, HHS said that the
state has met the SACWIS requirement in this area by building an interface between the
two systems.



Page 8                                                                      GAO-04-267T
Table 2: Selected SACWIS Child Welfare and Administrative Services

                                                           Fully or partially
                                                             operational in                      Planned
    Service                                                        SACWIS                    for SACWIS
    Child welfare services
    Child protectiona                                                        38                         5
                          b
    Out-of-home care                                                         35                         8
    Adoption                                                                 34                         9
    Independent living                                                       27                        14
    Intensive home-based servicesc                                           27                        13
    Administrative services
    Workload management                                                      32                         8
                      d
    IV-E eligibility                                                         29                        14
    Foster care maintenance payments                                         28                        14
    Adoption assistance payments                                             25                        17
    Contract provider payment                                                24                        15
Source: GAO survey.

Note: This table is based on responses from 46 states developing or operating a SACWIS. The rows
for the columns “fully or partially operational” and “planned” do not add to 46 because the
respondents may have answered “not supported,” “don’t know,” or “no answer.”
a
    Child protection includes services such as intake and screening, investigation, and disposition.
b
    Out-of-home care includes things such as foster care, group homes, and residential placement.
c
    Intensive home-based services include efforts to avoid placing a child in foster care.
d
    IV-E funding is available for foster care, adoption, and independent living services.



To assist with the design of their SACWIS, states relied on a number of
different participants, including internal users, such as caseworkers and
managers, information technology (IT) staff, and contractors. In
Oklahoma, for example, 150 child welfare staff from the field worked
closely with the contractor in intensive work group sessions to design and
test the system. To complement the caseworkers’ knowledge of child
welfare practice, 43 states relied on IT staff. Finally, 42 states reported that
they hired private contractors to conduct a large part of SACWIS design
and development.

At the time of our review, HHS reported that four states were not pursuing
SACWIS development, and most of these states reported various reasons
in our survey for not developing a system. In Hawaii, for example, the
child welfare agency chose not to pursue SACWIS because it already had a


Page 9                                                                                       GAO-04-267T
                         statewide system in place that it believed was adequately meeting its
                         needs and which was collecting and reporting federal child welfare data.


States Accrue Benefits   While most state child welfare agency officials said they recognize the
from Using SACWIS, but   benefits the state will achieve by developing SACWIS, such as enhancing
Several Issues Create    their ability to track the whereabouts of foster children, 31 state agencies
                         lag behind the time frames they set for completion, with 26 states
Delays in Completing     reporting delays ranging from 2 months to 8 years. According to survey
States’ Systems          results, automated systems provided easier access to data and allowed
                         caseworkers to better monitor children in their care, a fact that may
                         contribute to additional child welfare and administrative benefits, such as
                         decreased incidences of child abuse and neglect, shortened length of time
                         to achieve adoption, timeliness of payments to foster families, and
                         timeliness of payments to foster facilities. New Jersey, which is in the
                         planning stage, reported in our survey that its goal in developing a SACWIS
                         is to integrate the more than 40 stand-alone systems that currently capture
                         information on the children served by their child welfare agency.9 By
                         pulling all of these systems together into a uniform SACWIS, the state
                         hopes to improve the recording of casework activities in a timely manner
                         and to develop a tool to better target resources and services. Effectively
                         integrating these systems will require the state to use a disciplined IT
                         management approach that includes (1) detailed analyses of users’ needs
                         and requirements, (2) a clearly defined strategy for addressing information
                         needs, and (3) sufficient technical expertise and resources to support the
                         effort.

                         Despite the benefits that many states have accrued with SACWIS, 31 states
                         reported in our survey that they have been delayed in system completion
                         beyond their initial deadline and identified a number of challenges that
                         have led to the delay (see table 3).10 Some of the common difficulties states



                         9
                          New Jersey reported in our survey that it had spent approximately $9 million in federal
                         funds and $4 million in state and local funds on system development. According to HHS,
                         New Jersey first received federal funds in 1996.
                         10
                           Twelve of the 46 states reporting that they are developing or operating a SACWIS
                         reported that they have not experienced delays in developing their systems. In response to
                         the length of the delays reported by 26 states in our survey, ACF commented on a draft of
                         the July 2003 report that these states may be using different definitions in defining their
                         delays. However, ACF did not provide further information on how the delays represented
                         in that report differ from its perception of states’ experiences. In our survey, we asked
                         states to report on the delays that exceeded the time line outlined in their initial APD.



                         Page 10                                                                      GAO-04-267T
reported in developing SACWIS included receiving state funding approval,
reaching internal agreement on system development, and creating a
system that reflects child welfare work processes and is user-friendly.
Vermont officials, for example, reported that the state legislature declined
to provide the matching state funds needed to secure federal funding for
SACWIS. As a result, the state could not pursue development.

Table 3: Number of Months States Delayed in SACWIS Development

 State                                                              Length of delay in monthsa
 Alabama                                                                                           36
 Arkansas                                                                                           6
 California                                                                                        36
 Colorado                                                                                          26
 Connecticut                                                                                       96
 District of Columbia                                                                              36
 Georgia                                                                                           25
 Idaho                                                                                             21
 Illinois                                                                                          79
 Indiana                                                                                            6
 Kansas                                                                                            72
 Louisiana                                                                                         12
 Maryland                                                                                          12
 Michigan                                                                                          26
 Minnesota                                                                                         12
 Mississippi                                                                                       12
 New Jersey                                                                                        42
 New Mexico                                                                                         3
 Ohio                                                                                              36
 Oregon                                                                                            70
 Rhode Island                                                                                      14
 South Carolina                                                                                    47
 Tennessee                                                                                         36
 Utah                                                                                              48
 Virginia                                                                                           2
 Washington                                                                                        36
Source: GAO survey.

Note: While 31 states reported in the survey that they have experienced a delay in SACWIS
development, only 26 states reported the length of their delay. The survey was issued in October
2002 and completed by states as late as December 2002.



Page 11                                                                              GAO-04-267T
a
States were asked to report the number of months the delays exceeded the time line outlined in their
APD.
Despite user involvement in system design, some states still faced
challenges trying to reach internal agreement among agency officials and
caseworkers on the design of a system, resulting in a delay in
development. In New York—a state where the counties are responsible for
administering child welfare services—the development of SACWIS was
stalled when significant frustration with the system’s design led
commissioners from five large counties and New York City to request that
the state stop SACWIS development until a reassessment of the design of
and plans for the implementation of the system was completed.

Similarly, despite states’ heavy reliance on contractors, many reported that
securing contractors with knowledge of child welfare practice was a
challenge for timely SACWIS development. Contractors are hired by the
state for their system development knowledge but often are unfamiliar
with child welfare policies and practices, especially since they vary from
state to state. A contractor who has worked with seven states to develop
their SACWIS reported that contractors are asked to learn the child
welfare business practices of a state in a short amount of time and that
states cannot devote many resources, such as caseworkers, to help in the
design process because caseworkers need to devote their time to
providing services to children and families.

Many states reported that creating a system that reflects child welfare
work processes and is user-friendly was a challenge in developing
SACWIS. These issues were also identified in the federal reviews of states’
SACWIS. For example, one state explained in the SACWIS review that it
had designed a system to meet the caseworkers’ needs and reflect the
nature of the child welfare work processes by developing a system that
required events to be documented as they occurred. However, this design
limited the SACWIS’s functionality because it did not allow the
caseworkers to go back and enter information after an event happened.
The state explained that caseworkers do not use the system in real time,
but provide services to the children and families and then record the
information in the system. The state had to redesign the system to correct
for this design flaw.




Page 12                                                                             GAO-04-267T
HHS Provides Some           HHS has assisted states in a variety of ways in developing and completing
Assistance to Help States   their SACWIS.11 As a part of its regulatory responsibilities, HHS must
Meet SACWIS                 review, assess, and inspect the planning, design, development, installation,
                            and operation of SACWIS. In addition to reviewing and monitoring states’
Requirements                APDs, HHS conducts on-site SACWIS reviews to comply with these
                            responsibilities. HHS officials told us that these reviews are a detailed and
                            thorough assessment of state systems to ensure the systems’ compliance
                            with SACWIS requirements. In addition, officials reported that they
                            provide technical assistance during the on-site review to help states that
                            do not fully conform with the applicable regulations and policies. As of
                            October 2003, HHS had reviewed 27 SACWIS—5 of which were
                            determined as meeting all the requirements and classified as complete.
                            HHS officials told us that since states have the flexibility to build a
                            SACWIS that meets their needs, a large portion of the formal reviews
                            concentrate on ensuring that the systems conform to state business
                            practices. For example, while SACWIS regulations require that a state
                            report all AFCARS data from their SACWIS, one state HHS reviewed relied
                            on a separate state system to report data on the children served by the
                            juvenile justice agency who are eligible for IV-E foster care funds. The
                            state proved it had developed an automated process to merge data from
                            both systems to compile a single AFCARS report that included children
                            captured in both their SACWIS and juvenile justice systems. Therefore,
                            HHS recognized that this process best met the state’s needs and
                            determined the SACWIS to be complete and meeting all requirements.

                            Few systems have been determined complete after an on-site review
                            because of unresolved issues, such as not being able to build links to other
                            state information systems or not implementing certain eligibility
                            determination functions. To help states address some of these
                            development challenges, the SACWIS review team provides the state with
                            recommendations for complying with SACWIS requirements. In addition,
                            HHS officials reported that once the draft report with the results of the
                            SACWIS review is completed, federal staff schedule a conference call with




                            11
                             With regard to the budget difficulties that states reported facing, since 1994 the federal
                            government has made a commitment to help states develop and maintain their SACWIS by
                            matching 75 percent of states’ development funds through 1997 and providing an ongoing
                            match of 50 percent of state funding for the development and maintenance of their
                            systems. However, since the states’ legislatures must make the initial commitment to fund
                            SACWIS, the federal government cannot assist state child welfare agencies with this
                            challenge.



                            Page 13                                                                      GAO-04-267T
the state officials to walk through the system’s deficiencies and offer
guidance on how the state can move forward.

HHS facilitates the sharing of information between states developing
SACWIS through an automated system users’ group that allows state and
federal officials to exchange information, ideas, and concerns. In addition
to the users’ group, HHS officials also sponsor a Listserv—an electronic
mailing list—that allows state officials to exchange information and a
monthly conference call with state information technology directors. 12
Technical assistance for SACWIS development is also available to states
through the National Resource Center for Information Technology in Child
Welfare (Resource Center), which opened in 1999. According to survey
results, 9 states said they used the Resource Center for assistance in
developing SACWIS and 14 states reported using it for help with SACWIS
maintenance and improvements. According to Resource Center officials,
they assist states with SACWIS development by helping states understand
the technology that is available for use, providing information on the
automation of child welfare work and converting data, and reviewing the
APD documentation.




12
 In commenting on a draft of the July 2003 report, HHS indicated that a Web resource is
available to states interested in learning about other states’ efforts to develop human
services—child welfare, food stamps, Temporary Assistance to Needy Families, child care,
and child support enforcement—information systems at http://www.acf.hhs.gov/nhsitrc.




Page 14                                                                    GAO-04-267T
                            Several factors affect states’ ability to collect and report reliable13 data on
Several Factors Affect      children served by state child welfare agencies, and some problems exist,
the States’ Ability to      such as a lack of clear and documented guidance, with HHS’s oversight
                            and technical assistance. Almost all of the states responding to our survey
Ensure Reliable Data        reported that insufficient caseworker training and inaccurate and
on Children’s               incomplete data entry affect the quality of the data reported to HHS.14 In
                            addition, 36 of the 5015 states that responded to our survey reported that
Experiences, and            technical challenges, such as matching their state data element definitions
Some of HHS’s               to HHS’s data categories, affected the quality of the data that they report to
Oversight and               the federal government. Despite the assistance that HHS offers to states,
                            such as testing state data quality and providing the results to states to aid
Assistance Is               them in resubmitting data, states report ongoing challenges receiving clear
Problematic                 and documented guidance and obtaining technical assistance.


Insufficient Caseworker     Almost every state responding to our survey and all the states we visited
Training and Inaccurate     reported that insufficient training for caseworkers and inaccurate and
and Incomplete Data Entry   incomplete data entry affect the quality of the data reported to AFCARS
                            and NCANDS (see fig. 1). Although most states reported these as separate
Are the Most Common         factors, HHS and the states we visited found that insufficient training and
Factors That Affect Data    inaccurate and incomplete data entry are often linked. In official reviews
Reliability                 of states’ information systems’ capability to capture data and report them
                            to AFCARS, HHS advised states to offer additional training to caseworkers
                            on several AFCARS data elements, such as recording the reasons for a
                            child leaving foster care, to improve the accuracy of the data submitted.
                            However, state officials told us that training is typically one of the first
                            programs cut when states face tight budget restrictions. For example,
                            Iowa officials told us that training has been significantly reduced in recent
                            years because of budget cuts and new workers may wait 2 to 3 months
                            before being trained how to enter data appropriately into their SACWIS.


                            13
                             Data are reliable when they are complete and accurate. A subcategory of accuracy is
                            consistency. Consistency refers to the need to obtain and use data that are clear and well
                            defined enough to yield similar results in similar analysis. See U.S. General Accounting
                            Office, Assessing the Reliability of Computer-Processed Data, GAO-02-15G (Washington,
                            D.C.: Sept. 2002).
                            14
                             States were asked the extent to which certain problems may decrease the quality of the
                            data submitted to AFCARS and NCANDS using the following scale: very great, great,
                            moderate, some, and no affect.
                            15
                             The analysis of survey responses about reporting data to HHS is based on responses from
                            49 states and the District of Columbia. All states, regardless of SACWIS development, were
                            asked to complete these questions.



                            Page 15                                                                      GAO-04-267T
Figure 1: Most Common Caseworker Issues That Affect Data Quality

States reporting at least some affect in data quality
50       49                    49
                                                        46
                  45
                                                             43
                                       42
                              15
40       18      12
                                      11                17   12


30




20                            34
                 33
          31                          31                29   31


10




 0

        Inaccurate data       Insufficient          Incomplete data
        entry by              training for          entry by
        caseworkers           caseworkers           caseworkers



                  AFCARS

                  NCANDS

Source: GAO survey.

Notes: Based on responses from 50 states.

The results reported in the figure are a sum of the states that reported the issue had a very great
affect, great affect, moderate affect, or some affect on the quality of state data submitted to HHS.
Very great and great affect responses are represented in the top section of each bar. Moderate and
some affect responses are represented in the bottom section of each bar. States not included
answered “no affect,” “don’t know,” or “no answer.”



Inaccurate and incomplete data entry can also result from a number of
other factors, such as caseworkers’ hesitation to ask families for sensitive
information. For example, caseworkers in Oklahoma reported that they
did not feel comfortable asking if a child’s mother was married at the time
of birth or if a child is of Hispanic origin—both of which are required
AFCARS data elements. In commenting on a draft of this report, Oklahoma
added that caseworkers did not understand why the data elements were
required and how the federal government used the information. HHS noted




Page 16                                                                               GAO-04-267T
similar issues in five states that have had an AFCARS review.16
Caseworkers were inaccurately recording a child’s race as “unable to
determine” even though this option should be selected only if the child’s
parents or relatives cannot provide the information, such as when a child
is abandoned.17

Caseworkers, supervisors, and managers in the 5 states we visited
reported that additional factors, such as difficulties balancing data entry
with the time that they spend with the families and children, contributed
to inaccurate or incomplete data entry. Supervisors in Iowa explained that
since caseworkers are responsible for ensuring that children and their
families receive the services they need, the caseworkers tend to initially
limit data entry to the information that is necessary to ensure timely
payment to foster care providers and complete all other data elements
when they have time. In addition, caseworkers in Colorado said that they
are between 30 and 60 days behind in their data entry, so the information
in the automated system may not accurately reflect the current
circumstances of children in care. HHS’s Inspector General recently issued
a report in which more than two-thirds of the states reported that
caseworkers’ workloads, turnover, a lack of training, and untimely and
incomplete data entry affected the reporting of AFCARS data.18




16
 For the July 2003 report, we reviewed AFCARS reports from six of the eight states that
had been assessed by HHS—Arkansas, Connecticut, New Mexico, Texas, Vermont, and
Wyoming. HHS conducted reviews in Delaware and West Virginia after we completed our
analysis. As of October 2003, HHS had completed three additional reviews for North
Dakota, Rhode Island, and Washington.
17
   In commenting on a draft of the July 2003 report, ACF said that the finding from the
AFCARS reviews indicates that information is often defaulted to the response “unable to
determine” in order for the element not to fail the missing data standard, not that workers
are recording “unknown”; however, the report findings we used in this analysis instruct
states to fix the defaults and address caseworker practice by enhancing training on the
correct use of “unable to determine” when noting a child’s race.
18
 Department of Health and Human Services, Office of Inspector General, Adoption and
Foster Care Analysis and Reporting System (AFCARS): Challenges and Limitations,
OEI-07-01-00660 (Washington, D.C.: Mar. 2003).



Page 17                                                                       GAO-04-267T
Technical Challenges, such   In addition to data quality being affected by caseworker issues, many
as Matching State            states experienced technical challenges reporting their data to HHS. The
Definitions to Federal       problems reported by states are typically a result of challenges associated
                             with data “mapping”—matching state data elements to the federal data
Definitions, Affect Data     elements. For example, 36 states reported in our survey that matching
Reliability                  their state-defined data to HHS’s definitions affected the quality of the data
                             reported to NCANDS and AFCARS. Similarly, 24 states reported that
                             matching the more detailed data options available in their states’
                             information systems to the federal data elements affected the quality of
                             the data reported to NCANDS. Twenty-nine states reported that this issue
                             created challenges in reporting data to AFCARS. For example, following
                             an AFCARS assessment, HHS instructed a state that collects detailed
                             information on children’s disabilities, such as attention deficit disorder
                             and eating disorders, to map the information to the more limited options in
                             AFCARS, such as mental retardation and emotionally disturbed.

                             In many cases, states have to balance state policy with federal
                             requirements to ensure that they are reporting accurate data to AFCARS
                             and NCANDS, but are not contradicting their state policies. For example,
                             Texas officials reported that although the findings of their AFCARS review
                             instructed them to modify their SACWIS to collect, map, and extract data
                             on guardianship placements, the state does not support guardianship
                             arrangements.19 In addition, a recent report from the Child Welfare League
                             of America (CWLA) found that when reporting the number of times
                             children move from one foster care placement to another, states varied in
                             the type of placements included in that count.20 For example, 29 percent of
                             the states responding to CWLA’s survey included respite,21 25 percent
                             included runaways, and 16 percent included trial home visits when
                             reporting the number of placements a child had during the AFCARS report
                             period. According to federal guidance, the “number of placements”
                             element is meant to gather information on the number of times the child
                             welfare agency found it necessary to move a child while in foster care and
                             that by including runaways or trial home visits, a state is inflating the
                             number of moves a child experienced.


                             19
                              Guardianship arrangements occur when permanent legal custody of a child is awarded to
                             an individual, such as a relative, but the child is not legally adopted.
                             20
                              Child Welfare League of America. National Working Group Highlights, “Placement
                             Stability Measure and Diverse Out-of-Home Care Populations” (Washington, D.C., Apr.
                             2002).
                             21
                                  Respite care provides temporary child care for children away from their caretakers.



                             Page 18                                                                         GAO-04-267T
Although HHS Has Taken      HHS provides technical assistance for AFCARS and NCANDS reporting
Steps to Help States        through a number of resources. HHS officials in the central office and
Improve Their Data, Some    NCANDS contractor staff serve as the points of contact for states to ask
                            questions and seek guidance on reporting child welfare data. The officials
Problems with Its Efforts   in three of the five states that we visited said that the one-on-one focused
Exist                       technical assistance was useful when provided in a timely fashion. Most
                            state officials found the NCANDS data easier to report, in part because
                            more people were available for consultation and they were more
                            accessible and responsive. For example, states have access to four
                            NCANDS specialists and staff in the contractor’s central office when they
                            need assistance reporting child abuse and neglect information. However,
                            some of the states we visited reported that only one or two staff in HHS’s
                            central office are available to assist with AFCARS reporting.

                            In addition, the Resource Center offers states assistance with improving
                            data quality. However, Resource Center staff reported that the assistance
                            is geared more toward improving the limited data used in the federal
                            review process to monitor states’ compliance with child welfare laws and
                            federal outcome measures—CFSR—rather than all the data reported to
                            HHS. The Resource Center also sponsors an annual information
                            technology conference during which sessions covering all data-related
                            issues are held, including practices for ensuring data quality and outcome
                            evaluation in child welfare. In conjunction with this conference, the HHS
                            officials and the contractors that operate NCANDS hold an annual
                            technical assistance meeting for states to share ideas with one another,
                            discuss data elements that pose difficulties, and explore ways to address
                            these problems. In addition, an NCANDS state advisory group meets
                            annually to talk with HHS officials about NCANDS data and their
                            experiences reporting data. From these meetings, the state advisory group
                            proposes changes or improvements to NCANDS. HHS and state officials
                            reported that this partnership has helped ease some of the challenges in
                            reporting child abuse and neglect data.

                            HHS has also made available to states the software it uses to examine
                            states’ AFCARS and NCANDS submissions for inconsistencies and invalid
                            data. Officials in all the states we visited said that they regularly use this
                            software, and an HHS official said that nearly every state has used the
                            software at least once. When the data are submitted to HHS, they are run
                            through the same software, and HHS notifies the states of areas where
                            data are missing or inconsistent and allows the states to resubmit the data
                            after errors are corrected. HHS officials reported that these tests help
                            them to identify some data quality errors, such as missing data, and said
                            that they believe that, in general, data have improved in recent years.

                            Page 19                                                           GAO-04-267T
However, they indicated that the tests cannot pinpoint the underlying
problems contributing to these errors. Furthermore, one official reported
that no specific efforts have been conducted to track the individual data
elements, and therefore HHS cannot report on how data quality has
changed over time.

In an attempt to help states comply with the reporting standards and
address some of the factors that contribute to data quality problems, HHS
performs comprehensive reviews of state information systems’ ability to
capture AFCARS data to identify problems associated with data collection
and reporting and to ensure that the information in the automated system
correctly reflects children’s experiences in care. The assessments include
a technical review of the states’ computer code, a comparison of the data
from selected cases available in the information system to the case files,
and an improvement plan to resolve any errors. In addition, HHS officials
offer guidance to the states on improvements that can be made to the
information system and changes to program code used to report the
AFCARS data. HHS conducted pilot reviews in eight states between 1996
and 2000. By October 2003, HHS had conducted 11 official reviews—even
though states began reporting to AFCARS in 1995. According to results
from 6 of the 11 official AFCARS assessments we reviewed, no state met
the reporting requirements for all AFCARS data elements. The problems
noted in the reviews are similar to those of states responding to our survey
and those we visited. For example, most states received ratings of 2 or 3,
indicating technical and/or data entry errors that affect the AFCARS data
quality.22 For the current placement setting data element,23 for instance, 4
states received a rating of 2, 1 state received a rating of 3, and 1 state
received a rating of 4. In Connecticut, which received a rating of 2, HHS
found that, among other things, workers were not consistently entering
placement information in a timely way. It also found that workers entered



22
  HHS rates each data element using a four-point scale: (1) the AFCARS requirement(s) has
not been implemented in the information system; (2) the technical system requirements for
AFCARS reporting do not fully meet the standards; (3) the technical system requirements
for AFCARS reporting are in place, but there are data entry problems affecting the quality
of the data; (4) all of the AFCARS requirements have been met. According to an HHS
official, data elements that have a combination of technical and data entry problems are
rated as 2 until the technical issues are resolved. HHS will then rate the element as a 3 until
the data entry practices are changed.
23
  Current placement setting refers to a pre-adoptive home, foster family home-relative,
foster family home-nonrelative, group home, institution, supervised independent living,
runaway, or trial home visit.



Page 20                                                                         GAO-04-267T
placement data only into a narrative field, which resulted in placement
history gaps and incomplete AFCARS reports.

State officials in the six states for which we reviewed the HHS AFCARS
assessments reported that they found the reviews useful for improving
their AFCARS data submissions. In particular, they valued the thorough
review by HHS officials of the computer code states use to report the data.
Some of these officials reported that if all states were reviewed, the quality
of data available in AFCARS would improve tremendously. However, HHS
officials reported that they are not mandated to conduct the AFCARS
reviews and that priority is placed on other reviews, such as the CFSR and
SACWIS reviews. In addition, officials explained that the AFCARS reviews
are not conducted in states developing SACWIS until the systems are
operational. HHS expects to complete approximately four reviews each
year, depending on available resources, and has scheduled states through
2006. Similar to the SACWIS reviews, HHS officials offer recommendations
and technical assistance to states during the review on how they can
improve the quality of the data reported to AFCARS.

Although the states we visited appreciated some of HHS’s efforts to assist
with improving state data quality, they and most states responding to our
survey agreed that the assistance is not always consistent or easily
accessible (see fig. 2). States reported similar information to the Inspector
General—AFCARS data elements were not clearly and consistently
defined and technical assistance is effective but difficult to access.




Page 21                                                          GAO-04-267T
Figure 2: Federal Practices That Affect Data Quality

States reporting at least some affect in data quality
50
      45

                             41
40
       17       35                                 35
                             11
                                                   6
30             9
                                     25

                                     5
20
                                                         17
                             30                    29
       28                                                4
               26
                                    20
10
                                                         13


 0

       Imprecise           Lack of clear,          Difficulty
     definitions for       documented             accessing
     required fields         guidance              technical
                                                  assistance
                                                 and guidance

              AFCARS

              NCANDS

Source: GAO survey.

Notes: Based on responses from 50 states.

The results reported in the figure are a sum of the states that reported the issue had a very great
affect, great affect, moderate affect, or some affect on the quality of state data submitted to HHS.
Very great and great affect responses are represented in the top section of each bar. Moderate and
some affect responses are represented in the bottom section of each bar. States not included
answered “no affect,” “don’t know,” or “no answer.”



The primary concerns reported by the states we visited were delays in
receiving clear written guidance on defining and reporting certain data
elements and the lack of state input in suggesting changes to AFCARS.
Despite the written guidance available to states in the form of regulations
and an online policy manual, states reported that the variation in state
policies and practices makes it difficult to interpret how to apply the
general guidance. As a result, states consult with HHS to ensure they are
applying the regulations appropriately. However, in commenting on a draft
of this report, officials in Oklahoma told us that a common concern among
the states is the lack of timely response from HHS when seeking guidance


Page 22                                                                               GAO-04-267T
                       on how to report data. In addition, officials in New York explained they
                       have made it a practice to check the HHS Web site on a regular basis for
                       current guidance but have not found it a useful tool, and may turn to other
                       states for guidance on AFCARS reporting. In commenting on a draft of this
                       report, HHS explained that it first refers states to its Web site for
                       information and believes that the available guidance addresses states’
                       concerns in most instances. In addition, the states that have had an
                       AFCARS review experienced delays in obtaining guidance on how to
                       proceed following the on-site review. For example, Texas officials
                       reported that the state sought clarification on its improvement plan and
                       submitted additional questions to HHS following the review. However,
                       when we spoke with the state officials, they said that they had been
                       waiting 3 months for a response on how to proceed. An HHS official told
                       us that since the review process is relatively new, the agency is still
                       developing a process to respond to the states and recognizes that it has
                       not been responsive to the states already reviewed. In addition, HHS is
                       taking steps to gather feedback from states and other users of AFCARS
                       data to determine how to improve the system to make the data more
                       accurate and usable. As a part of these efforts, HHS has published a
                       Federal Register notice soliciting comments and held focus group
                       meetings at national conferences. The difficulties states face in receiving
                       federal guidance and assistance, as well as the other challenges they face
                       in reporting data, may negatively affect the reliability of the data available
                       in AFCARS and NCANDS.


                       Some states are using a variety of practices to address the challenges
States Are Using       associated with developing SACWIS and improving data reliability,
Various Practices to   although no formal evaluations of their effectiveness are available. To
                       address the challenge of developing a system to meet statewide needs,
Overcome System        states relied on caseworkers and supervisors from local offices to assist in
Development            the design and testing of the system. Few states reported in our survey
                       strategies to overcome the other key challenges, such as limited funding
Challenges and         and the difficulty of securing knowledgeable contractors, but some states
Improve Data on        we visited have devised some useful approaches. To improve data
Children’s             reliability, the five states we visited routinely review their data to identify
                       data entry errors so that managers can ensure that the missing data are
Experiences            entered appropriately.




                       Page 23                                                            GAO-04-267T
States Are Primarily       To overcome development challenges, survey respondents emphasized the
Relying on SACWIS Users    importance of including system users in the various phases of completing
to Overcome Some of the    SACWIS—planning, design, development, testing, and implementation.
                           Past GAO work and other research efforts have determined similar
Challenges to Completing   approaches as best practices in building information systems.24 Forty-four
Their Systems              of the 46 states responding to our survey that they are developing or
                           operating a SACWIS indicated that they relied on internal users, such as
                           caseworkers and supervisors, in the development of their systems and 34
                           of these states said that they were extremely helpful participants. The
                           extent to which the users were involved in development differed across
                           the states. For example, in Texas, caseworkers from all of their child
                           welfare regions were recruited to provide input on design and
                           development, as well as during initial testing, pilot testing, and
                           implementation of the system. Arkansas reported establishing a committee
                           made up of users to review the work plan and sign off on recommended
                           changes.

                           Ten states noted that user input should not be limited to frontline workers,
                           such as caseworkers, but should include representatives from other areas
                           of the agency, such as the financial staff, and other agencies that serve
                           children, such as child support enforcement.25 While not one of the most
                           common challenges reported in our survey, New Hampshire reported that
                           one of its challenges with meeting its SACWIS timeframe was not working
                           collaboratively with other agencies, such as Temporary Assistance for
                           Needy Families (TANF)26 and child support enforcement, to develop the
                           payment component of SACWIS. To attempt to overcome this challenge,
                           26 of the 46 states responding to our survey that they are developing or
                           operating a SACWIS indicated that they included external public agency


                           24
                            See U.S. General Accounting Office, Executive Guide: Improving Mission Performance
                           Through Strategic Information Management and Technology, GAO/AIMD-94-115
                           (Washington, D.C.: May 1, 1994); Center for Technology in Government, University of
                           Albany, SUNY. Tying a Sensible Knot: A Practical Guide to State-Local Information
                           Systems. Albany, N.Y., June 1997.
                           25
                             The Child Support Enforcement Program is a joint federal, state, and local partnership
                           that was established in 1975 under Title IV-D of the Social Security Act. Each state runs a
                           child support program, which provides four major services: locating noncustodial parents,
                           establishing paternity, establishing child support obligations, and collecting child support
                           for families.
                           26
                            In 1996, the Congress created the block grant Temporary Assistance for Needy Families
                           program replacing the Aid to Families with Dependent Children (AFDC) and related
                           welfare programs. States were given increased flexibility in designing the eligibility criteria
                           and benefit rules, which require work in exchange for time-limited benefits.



                           Page 24                                                                         GAO-04-267T
                               users, and 23 reported using representatives from other state agencies that
                               serve children in developing their SACWIS.

                               In addition to seeking input from caseworkers and other system users
                               while developing SACWIS, many states continue to include users as a part
                               of the implementation teams, to serve as contacts in the field and provide
                               ongoing assistance, and to provide input on system enhancements.
                               Alabama responded in our survey that the state had “mentors” in each
                               county to help caseworkers adjust to the new system. These mentors
                               continue to provide ongoing support now that the system is implemented.
                               Oklahoma recruits experienced child welfare field staff for its SACWIS
                               help desk because of their knowledge of the system and child welfare
                               policy and practice.

                               Although states faced other challenges in completing their SACWIS, few
                               reported implementing approaches to overcome the barriers. According to
                               survey results, a common problem states faced in developing SACWIS was
                               receiving insufficient state funding for development. States did not report
                               in our survey, however, approaches for obtaining more funding for
                               developing SACWIS, and few states reported developing strategies in an
                               attempt to overcome the challenges associated with tight budgets for
                               maintaining their systems. For example, Iowa officials engaged in careful
                               planning with system users to ensure that they addressed the highest
                               priorities when enhancing the system. In particular, the officials reported
                               that maintaining tight control over the development and maintenance
                               processes helps them avoid investing inordinate amounts of resources to
                               make corrections to the system. Similarly, few states reported on
                               approaches to overcome the challenge of finding contractors with
                               knowledge of child welfare practice. However, Iowa officials explained
                               that once the contract staff are hired, they are required to attend the same
                               training as new caseworkers to ensure that they are familiar with the
                               state’s child welfare policies and to familiarize themselves with casework
                               practices.


States Use Strategies, such    Twenty-eight states reported using approaches to help caseworkers
as Producing Reports That      identify the data elements that are required for federal reporting and to
Identify Missing Data, in an   help them better understand the importance of entering timely and
                               accurate data. Ten states responding to our survey reported reviewing the
Attempt to Improve the         federal reporting requirements in training sessions as a way to improve
Reliability of the Data        data quality. For example, Tennessee reported that the state added a
Reported to HHS                component about AFCARS to the initial and ongoing training workers
                               receive about using SACWIS. The curriculum addresses the AFCARS

                               Page 25                                                         GAO-04-267T
report in general and the individual data elements to help the caseworkers
better understand the purpose of collecting the information. In Nebraska,
a “desk aid” that explains the data elements and where and why to enter
them in the system is available on the caseworkers’ computer desktops. In
addition, New York has developed a step-by-step guide explaining to
workers how NCANDS data should be entered, with references to the
policy or statute requiring the information.

To improve data reliability, some states have designed their information
systems with special features to encourage caseworkers to enter the
information. Four states responding to our survey and three states we
visited designed their SACWIS with color-coded fields to draw attention to
the data elements that caseworkers are required to enter. Colorado, Iowa,
New York, and Oklahoma have built into their systems alerts—also known
as “ticklers”—to remind caseworkers and supervisors of tasks that they
need to complete. For example, in Oklahoma, a stoplight icon on the
caseworker’s computer desktop reminds the worker when tasks are due. A
green light indicates that nothing is due within 5 days; a yellow light means
that something is due within 5 days; and a red light means that something
is overdue. Caseworkers and supervisors in the states we visited had
mixed responses about the usefulness and effectiveness of the alerts.
Some caseworkers found them to be a nuisance, while other caseworkers
and supervisors found them to be useful tools in managing workloads and
prioritizing daily tasks.

Six states reported that the best way to improve data quality was to use
the data in published reports and hold the caseworkers and supervisors
accountable for the outcomes of the children in their care. In addition, six
states responding to our survey reported using the data available in their
information systems to measure state outcomes similar to the CFSR. State
officials reported that this approach is an effective way to get local offices
invested in the quality of the data. For example, North Carolina publishes
monthly reports for each county comparing their performance on state
data indicators, such as the length of time children spend in care, to
counties of similar size and the state as a whole. County officials reported
that these reports encourage workers to improve the quality of the data
collected and entered into the state system since their performance is
being widely published and compared with that of other counties.

In addition, all the states we visited reported that frequent review of their
data, such as using software from HHS to test their AFCARS and NCANDS
data to pinpoint data entry errors prior to submitting them to HHS, has
helped improve data quality. When the states identify poor data, they alert

Page 26                                                           GAO-04-267T
               the caseworkers and supervisors of needed corrections and data entry
               improvements. For example, Colorado runs these reports about four to
               five times a year, with one run occurring approximately 6 weeks before
               each AFCARS submission. When the data specialists find errors, they
               notify the caseworker to clean up the data.


               While most states are developing statewide information systems,
Concluding     challenges with data reliability remain. Although SACWIS development is
Observations   delayed in many states, state officials recognize the benefits of having a
               uniform system that enhances the states’ ability to monitor the services
               provided and the outcomes for children in their care. Although states
               began reporting to NCANDS in 1990 and were mandated to begin reporting
               to AFCARS in 1995, most states continue to face challenges providing
               complete, accurate, and consistent data to HHS. In addition, the results of
               more recent HHS efforts, such as conducting AFCARS-related focus
               groups, are unknown. Reliable data are essential to the federal
               government’s development of policies that address the needs of the
               children served by state child welfare agencies and its ability to assist
               states in improving child welfare system deficiencies. Without well-
               documented, clearer guidance and the completion of more comprehensive
               reviews of states’ AFCARS reporting capabilities, states are limited in
               overcoming challenges that affect data reliability. Because these
               challenges still remain, HHS may be using some questionable data as the
               foundation for national reports and may not have a clear picture of how
               states meet the needs of children in their care.

               To improve the reliability of state-reported child welfare data, we
               recommended in our July 2003 report that the Secretary of HHS consider,
               in addition to HHS’s recent efforts to improve AFCARS data, ways to
               enhance the guidance and assistance offered to states to help them
               overcome the key challenges in collecting and reporting child welfare
               data. These efforts could include a stronger emphasis placed on
               conducting AFCARS reviews and more timely follow-up to help states
               implement their improvement plans or identifying a useful method to
               provide clear and consistent guidance on AFCARS and NCANDS
               reporting. ACF generally agreed with our findings and commented that the
               report provides a useful perspective of the problems states face in
               collecting data and of ACF’s effort to provide ongoing technical assistance
               to improve the quality of child welfare data. In response to our
               recommendation, ACF said that we categorized its efforts as “recent” and
               did not recognize the long-term efforts to provide AFCARS- and NCANDS-
               related guidance to the states. Although we did not discuss each effort in

               Page 27                                                        GAO-04-267T
                  depth, we did mention the agency’s ongoing efforts in our report. ACF also
                  noted in its comments that the data definitions need to be updated and
                  revised and said it is currently in the process of revising the AFCARS
                  regulations to further standardize the information states are to report—
                  which we acknowledged in our report. ACF also commented that it is
                  firmly committed to continue to support the states and to provide
                  technical assistance and other guidance as its resources will permit. ACF
                  commented that it provided increased funding to the National Resource
                  Centers in fiscal year 2003, and it believed that this increase will improve
                  ACF’s ability to provide assistance to the states. After receiving the draft
                  report for comment, HHS separately provided information on an
                  additional service the National Resource Center for Information
                  Technology in Child Welfare provides to states. More recently, HHS said
                  that it would be creating policy guidance that will delineate what will
                  happen if a state fails to complete its SACWIS within a reasonable time
                  frame. For example, funding may become contingent on successful
                  completion of specific milestones.


                  Mr. Chairman, this concludes my prepared statement. I would be pleased
                  to respond to any questions that you or other members of the
                  Subcommittee may have.


                  For further contacts regarding this testimony, please call Cornelia M.
GAO Contact and   Ashby at (202) 512-8403. Individuals making key contributions to this
Acknowledgments   testimony include Diana Pietrowiak and Sara Schibanoff.




                  Page 28                                                         GAO-04-267T
Related GAO Products


             Child Welfare: Enhanced Federal Oversight of Title IV-B Could Provide
             States Additional Information to Improve Services. GAO-03-956.
             Washington, D.C.: September 12, 2003.

             Child Welfare: Most States Are Developing Statewide Information
             Systems, but the Reliability of Child Welfare Data Could Be Improved.
             GAO-03-809. Washington, D.C.: July 31, 2003.

             Child Welfare and Juvenile Justice: Several Factors Influence the
             Placement of Children Solely to Obtain Mental Health Services.
             GAO-03-865T. Washington, D.C.: July 17, 2003.

             Child Welfare and Juvenile Justice: Federal Agencies Could Play a
             Stronger Role in Helping States Reduce the Number of Children Placed
             Solely to Obtain Mental Health Services. GAO-03-397. Washington, D.C.:
             April 21, 2003.

             Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare
             Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March
             31, 2003.

             Human Services: Federal Approval and Funding Processes for States’
             Information Systems. GAO-02-347T. Washington, D.C.: July 9, 2002.

             Foster Care: Recent Legislation Helps States Focus on Finding
             Permanent Homes for Children, but Long-Standing Barriers Remain.
             GAO-02-585. Washington, D.C.: June 28, 2002.

             Human Services Integration: Results of a GAO Cosponsored Conference
             on Modernizing Information Systems. GAO-02-121. Washington, D.C.:
             January 31, 2002.

             District of Columbia Child Welfare: Long-Term Challenges to Ensuring
             Children’s Well-Being. GAO-01-191. Washington, D.C.: December 29, 2000.

             Child Welfare: New Financing and Service Strategies Hold Promise, but
             Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.

             Welfare Reform: Improving State Automated Systems Requires
             Coordinated Federal Effort. GAO/HEHS-00-48. Washington, D.C.: April 27,
             2000.




             Page 29                                                      GAO-04-267T
           Foster Care: States’ Early Experiences Implementing the Adoption and
           Safe Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.

           Foster Care: HHS Could Better Facilitate the Interjurisdictional
           Adoption Process. GAO/HEHS-00-12. Washington, D.C.: November 19,
           1999.

           Foster Care: Effectiveness of Independent Living Services Unknown.
           GAO/HEHS-00-13. Washington, D.C.: November 5, 1999.

           Foster Care: Kinship Care Quality and Permanency Issues.
           GAO/HEHS-99-32. Washington, D.C.: May 6, 1999.

           Juvenile Courts: Reforms Aim to Better Serve Maltreated Children.
           GAO/HEHS-99-13. Washington, D.C.: January 11, 1999.

           Child Welfare: Early Experiences Implementing a Managed Care
           Approach. GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.

           Foster Care: Agencies Face Challenges Securing Stable Homes for
           Children of Substance Abusers. GAO/ HEHS-98-182. Washington, D.C.:
           September 30, 1998.

           Managing Technology: Best Practices Can Improve Performance and
           Produce Results. GAO/T-AIMD-97-38, January 31, 1997.

           Child Welfare: HHS Begins to Assume Leadership to Implement National
           and State Systems. GAO/AIMD-94-37. Washington, D.C.: June 8, 1994.

           Executive Guide: Improving Mission Performance Through Strategic
           Information Management and Technology. GAO/AIMD-94-115.
           Washington, D.C.: May 1, 1994.




(130344)
           Page 30                                                     GAO-04-267T
This is a work of the U.S. government and is not subject to copyright protection in the
United States. It may be reproduced and distributed in its entirety without further
permission from GAO. However, because this work may contain copyrighted images or
other material, permission from the copyright holder may be necessary if you wish to
reproduce this material separately.
                         The General Accounting Office, the audit, evaluation and investigative arm of
GAO’s Mission            Congress, exists to support Congress in meeting its constitutional responsibilities
                         and to help improve the performance and accountability of the federal
                         government for the American people. GAO examines the use of public funds;
                         evaluates federal programs and policies; and provides analyses,
                         recommendations, and other assistance to help Congress make informed
                         oversight, policy, and funding decisions. GAO’s commitment to good government
                         is reflected in its core values of accountability, integrity, and reliability.


                         The fastest and easiest way to obtain copies of GAO documents at no cost is
Obtaining Copies of      through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-
GAO Reports and          text files of current reports and testimony and an expanding archive of older
                         products. The Web site features a search engine to help you locate documents
Testimony                using key words and phrases. You can print these documents in their entirety,
                         including charts and other graphics.
                         Each day, GAO issues a list of newly released reports, testimony, and
                         correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site
                         daily. The list contains links to the full-text document files. To have GAO e-mail
                         this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail
                         alerts” under the “Order GAO Products” heading.


Order by Mail or Phone   The first copy of each printed report is free. Additional copies are $2 each. A
                         check or money order should be made out to the Superintendent of Documents.
                         GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a
                         single address are discounted 25 percent. Orders should be sent to:
                         U.S. General Accounting Office
                         441 G Street NW, Room LM
                         Washington, D.C. 20548
                         To order by Phone:     Voice:    (202) 512-6000
                                                TDD:      (202) 512-2537
                                                Fax:      (202) 512-6061


                         Contact:
To Report Fraud,
                         Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in      E-mail: fraudnet@gao.gov
Federal Programs         Automated answering system: (800) 424-5454 or (202) 512-7470


                         Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800
Public Affairs           U.S. General Accounting Office, 441 G Street NW, Room 7149
                         Washington, D.C. 20548