oversight

OIG's Independent Report on the Department's Performance Summary Report for Fiscal Year 2007

Published by the Department of Education, Office of Inspector General on 2008-02-01.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                                    UNITED STATES DEPARTMENT OF EDUCATION
                                           OFFICE OF INSPECTOR GENERAL



                                                               February 1, 2008

MEMORANDUM


To:       	          Deborah Price
                     Assistant Deputy Secretary
                     Office of Safe and Drug-Free Schools


From:	               Keith West /s/
                     Assistant Inspector General for Audit Services

Subject: 	           Office of Inspector General’s Independent Report on the U.S. Department of
                     Education’s Performance Summary Report for Fiscal Year 2007, dated January
                     30, 2008.

Attached is our authentication of management’s assertions contained in the U.S. Department of
Education’s Performance Summary Report for Fiscal Year 2007, dated January 30, 2008, as
required by section 705(d) of the Office of National Drug Control Policy (ONDCP)
Reauthorization Act of 1998 (21 U.S.C. § 1704(d)).

Our authentication was conducted in accordance with the guidelines stated in the Office of
National Drug Control Policy Circular: Drug Control Accounting, dated May 1, 2007.

If you have any questions or wish to discuss the contents of this authentication, please contact
Michele Weaver-Dugan, Director, Operations Internal Audit Team, at (202) 245-6941.


                                                             Attachment




 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
                                    UNITED STATES DEPARTMENT OF EDUCATION
                                           OFFICE OF INSPECTOR GENERAL



                                                         February 1, 2008


Office of Inspector General’s Independent Report on the U.S. Department of Education’s
Performance Summary Report for Fiscal Year 2007, dated January 30, 2008.

We have reviewed management’s assertions contained in the accompanying Performance
Summary Report, titled Department of Education Performance Summary Report for Fiscal Year
2007, dated January 30, 2008. The U.S. Department of Education’s management is responsible
for the Performance Summary Report and the assertions contained therein.

Our review was conducted in accordance with attestation standards established by the American
Institute of Certified Public Accountants. A review is substantially less in scope than an
examination, the objective of which is the expression of an opinion on management’s assertions.
Accordingly, we do not express such an opinion.

We performed review procedures on the “Performance Summary Information,” “Assertions,”
and “Criteria for Assertions” contained in the accompanying Performance Summary Report. In
general, our review procedures were limited to inquiries and analytical procedures appropriate
for our review engagement. We did not perform procedures related to controls over the reporting
system noted in the attached report.

Based on our review, nothing came to our attention that caused us to believe that management’s
assertions, contained in the accompanying Performance Summary Report, are not fairly stated in
all material respects, based upon the Office of National Drug Control Policy Circular: Drug
Control Accounting, dated May 1, 2007.




                                                                   Keith West /s/       

                                                                   Assistant Inspector General for Audit Services 





 The Department of Education's mission is to promote student achievement and preparation for global competitiveness by fostering educational
                                                   excellence and ensuring equal access.
       Department of Education 





         Performance Summary Report 


                 Fiscal Year 2007 





                   In Support of the

            National Drug Control Strategy 


As required by ONDCP Circular: Drug Control Accounting




                   January 30, 2008
                                             Department of Education 


                         Performance Summary Report for Fiscal Year 2007 


                                               TABLE OF CONTENTS


Transmittal Letter ............................................................................................................ 1


Performance Summary Information................................................................................. 2 


          Safe Schools/Healthy Students ............................................................................ 2 


          Student Drug Testing............................................................................................ 3 


          Safe and Drug-Free Schools and Communities State Grants Program................ 7 


          Grants to Reduce Alcohol Abuse........................................................................ 13 


Assertions ..................................................................................................................... 17 


          Performance Reporting System.......................................................................... 17 


          Methodology for Establishing Performance Targets ………………………………17 


          Performance Measures for Significant Drug Control Activities ........................... 18 


Criteria for Assertions ................................................................................................... 18 

Mr. John P. Higgins, Jr.
Inspector General
U.S. Department of Education
400 Maryland Avenue, S.W.
Washington, DC 20202-1510

Dear Mr. Higgins:

As required by Office of National Drug Control Policy (ONDCP) Circular Drug Control
Accounting, enclosed please find detailed information about performance-related
measures for key drug control programs administered by the U.S. Department of
Education, in accordance with the guidelines in the circular dated May 1, 2007. This
information corresponds to the Safe and Drug-Free Schools and Communities program,
which is the Department’s only Drug Control Budget Decision Unit displayed in the
National Drug Control Budget Summary.

Consistent with the instructions in the ONDCP Circular, please provide your
authentication to me in writing and I will transmit it to ONDCP along with the enclosed
Performance Summary Report. As you know, ONDCP requests these documents by
February 1, 2008 if possible. Please do not hesitate to contact me if you have any
questions about the enclosed information.

                                  Sincerely,




                                  Deborah A. Price
                                  Assistant Deputy Secretary for
                                  Office of Safe and Drug-Free Schools




                                            1

Performance Summary Information

                             Safe Schools/Healthy Students

Measure 1: The percentage of grantees demonstrating a decrease in substance abuse
over the three-year grant period (Safe Schools/Healthy Students – FY 2004 and FY
2005 Cohorts)

Table 1

Cohort FY 2003 FY 2004 FY 2005 FY 2006 FY 2007 FY 2007 FY 2008
        Actual  Actual  Actual  Actual  Target  Actual  Target
2004   n/a     n/a     n/a     75      90      pending none
2005   n/a     n/a     n/a     n/a     none    pending 86.25

The measure. This performance measure is for the Safe Schools/Healthy Students
initiative, a joint project of the Departments of Education, Health and Human Services,
and Justice. The initiative provides grants to local educational agencies (LEAs) to
support the development and implementation of a comprehensive plan designed to
prevent student drug use and violence and support healthy youth development.

This measure, one of four for this initiative, focuses on one of the primary purposes of
the initiative – reduced student drug use. The initiative, and this measure, is directly
related to the National Drug Control Strategy’s goal of preventing drug use before it
begins. Grantees select and report on one or more measures of prevalence of drug use
for students. For the FY 2004 – 2006 cohorts, those grantee measures are not
common across grant sites but, rather, reflect priority drug use problems identified by
sites.

FY 2007 Performance Results. Because the measure is established to look at progress
over the three-year grant period, it has just a single target for the third year of
implementation of each cohort. Sites were not required to provide or collect baseline
data at the time of application or before program interventions were implemented, so
grantees provided baseline data for their selected measures related to drug use after
year one (for example in FY 2005 for the FY 2004 cohort). Grantees from the FY 2004
cohort provided data in late November 2007 as part of a semi-annual performance
report. Those data will be aggregated later in FY 2008 to determine if the FY 2007
target for the cohort has been met. Interim data for the 2005 cohort will also be
reported on the same approximate time schedule.

FY 2008 Performance Targets. The FY 2005 cohort will be providing data for this
measure near the end of 2008. The 2007 target for this cohort was set based on the
results from the FY 2004 cohort; we calculated an increase of 15 percentage points in
terms of grantees demonstrating decreased substance abuse in year two of the FY
2004 cohort. Because GPRA measures for this initiative were first implemented for the
FY 2004 cohort, targets for this initial cohort represented our judgment at the time, given


                                            2

the significant size of Safe Schools/Healthy Students grants and the emphasis on
research-based programs that is central to the initiative. We elected to revise the target
for the FY 2005 cohort for the measure based on the actual performance to date
(implementation year two) of the FY 2004 cohort. Based on our professional judgment,
it seemed that the revised target of 86.25 percent was appropriately aggressive and that
attaining that target would be a meaningful outcome for the program, while
acknowledging that our target for the initial (FY 2004) cohort may have been unrealistic.

Methodology. Data are collected by grantees, generally using student surveys. Data
are furnished in the second of two semi-annual performance reports provided by
grantees each project year. If grantees identified more than one measure of drug
abuse, or provided data for individual school-building types (for example, separate data
for middle and high schools), grantees were considered to have experienced a
decrease in substance abuse if data for a majority of measures provided reflected a
decrease. If a grant site provided data for an even number of measures and half of
those measures reflected a decrease and half reflected no change or an increase, that
grant site was judged not to have demonstrated a decrease in substance abuse. The
response rate for the FY 2004 cohort for this measure was 35 percent. While most sites
were able to provide some data related to this measure, we considered as valid data
only data from sites that used the same elements/items in each of years one and two.

If data for this measure are not available at the time that performance reports are
submitted, staff follow up with sites to attempt to obtain data for the measure. Grantees
that fail to provide data are not included in the tabulation of data for the measures.
Also, grantees that did not provide data for two consecutive project years (so that we
could determine if a decrease in substance abuse had occurred) are not included in
data report for the measure. Authorized representatives for the grant site sign the
annual performance report and, in doing so, certify that to the best of the signers’
knowledge and belief, all data in the performance report are true and correct and that
the report fully discloses all known weaknesses concerning the accuracy, reliability, and
completeness of the data included. Generally, the Department relies on the certification
concerning data supplied by grantees and does not conduct further reviews.

Targets were established for this measure after the baseline data for the FY 2004
cohort were provided. Based on the final results for this first cohort, targets for future
cohorts may need to be adjusted. (For example, the target for the FY 2005 cohort was
already adjusted.)

                                  Student Drug Testing

Measure 2: The percentage of student drug testing grantees that experience a 5
percent reduction in current (30-day) illegal drug use by students in the target
population. (Student Drug Testing – FY 2003 and FY 2005 cohorts)




                                             3

Table 2

Cohort FY 2003 FY 2004 FY 2005 FY 2006 FY 2007 FY 2007 FY 2008
        Actual  Actual  Actual  Actual  Target  Actual  Target
2003   n/a     n/a     n/a     33      50      pending n/a
2005   n/a     n/a     n/a     n/a     33      pending 50

The measure. This measure is one of two measures for the Student Drug-Testing
Programs grant competition. The competition provides discretionary grants to LEAs,
community-based organizations, or other public and private entities to support
implementation of drug testing of students, consistent with the parameters established
by the U.S. Supreme Court or for students and their families that voluntarily agree to
participate in the student drug testing program.

This measure is directly related to the National Drug Control Strategy’s goal related to
preventing drug use before it starts. Student drug testing has been prominently
featured in recent annual versions of the strategy as a recommended drug prevention
intervention.

FY 2007 Performance Results. Data for the FY 2003 cohort will be submitted as part of
final reports for these grants. The grantees are currently operating under no-cost
extensions; final reports were due at the end of 2007, and aggregate data will be
available in March 2008.

We have completed a preliminary review of data submitted by the FY 2005 cohort for
this measure and have identified significant concerns about the quality and
comparability of the data. Grant sites have reported on prevalence rates for a variety of
illegal drugs and have not always provided data from the same items/elements across
project years one and two. Based on these concerns, we will be obtaining assistance
from the U.S. Department of Education’s Data Quality Initiative contractor to create and
disseminate detailed data collection and reporting guidance for the program, as well as
data standards that we will use to determine what constitutes valid data for this
measure. After that project is complete in 2008, we will aggregate and report data for
the FY 2005 cohort based on these standards and report it.

FY 2008 Performance Targets. We established targets for the percentage of grantees
experiencing a 5 percent reduction in current illegal drug use after reviewing the first two
years of data for the FY 2003 cohort of grant sites. Consistent with research that
suggests that changes in student behavior related to student drug testing may not be
realized immediately, we assumed that we could look for an increased number of
grantees to experience positive change and, using our professional judgment, set that
target at 50 percent of grantees. When we have received data for three project years
from a single cohort of sites, we will revisit targets for future cohorts. We may also need
to consider revising targets for future cohorts based on full results from the FY 2005
cohort because the FY 2003 cohort is very small (only 8 sites) and may not be typical of
other cohorts.


                                             4

Methodology Data are collected by grantees using student surveys. Data are provided
as part of the grantees’ annual performance reports. Grantees do not use the same
survey items to collect data for this measure but, rather, self-select survey items (often
from surveys already administered) in order to provide these data. Survey items may
relate to different substances, but must collect information concerning current use in
order to be included in the data reported for this measure. Grantees did not provide
baseline data in their applications, so we have to wait until grantees provide data from
project year one and two in order to determine if they have experienced a decrease in
substance abuse. For the FY 2003 cohort, project implementation was delayed for one
full year while grantees sought needed institutional review board clearance to drug test
students, so performance data were received in 2005 and 2006. Only 3 of 8 grantees
provided comparable data across the first two years of their project.

Authorized representatives for the grant site sign the annual performance report and, in
doing so, certify that to the best of the signer’s knowledge and belief, all data in the
performance report are true and correct and that the report fully discloses all known
weaknesses concerning the accuracy, reliability, and completeness of the data
included. Generally, the Department relies on the certification concerning data supplied
by grantees and does not conduct further reviews.

No new grants were awarded under this program in FY 2004.

The anticipated levels of decrease in substance abuse are consistent with those
included in the National Drug Control Strategy – five percent per year. Targets were
established following the report of baseline data for grant sites from the FY 2003 cohort.

Measure 3: The percentage of student drug testing grantees that experience a 5
percent reduction in past-year illegal drug use by students in the target population.
(Student Drug Testing – FY 2003 and FY 2005 cohort)

Table 3

Cohort FY 2003 FY 2004 FY 2005 FY 2006 FY 2007 FY 2007 FY 2008
        Actual  Actual  Actual  Actual  Target  Actual  Target
2003   n/a     n/a     n/a     25      50      Pending n/a
2005   n/a     n/a     n/a     n/a     25      Pending 50

The measure. This measure is one of two measures for the Student Drug-Testing
Programs grant competition. The competition provides discretionary grants to LEAs,
community-based organizations, or other public and private entities to support
implementation of drug testing of students, consistent with the parameters established
by the U.S. Supreme Court or for students and their families that voluntarily agree to
participate in the student drug testing program.

This measure is directly related to the National Drug Control Strategy’s goal related to
preventing drug use before it starts. Student drug testing has been prominently


                                            5

featured in recent annual versions of the strategy as a recommended drug prevention
intervention.

FY 2007 Performance Results. Data for the FY 2003 cohort will be submitted as part of
final reports for these grants. The grantees are currently operating under no-cost
extensions; final reports were due at the end of 2007, and aggregate data will be
available in March 2008.

We have completed a preliminary review of data submitted by the FY 2005 cohort for
this measure and have identified significant concerns about the quality and
comparability of the data. Grant sites have reported on prevalence rates for a variety of
illegal drugs and have not always provided data from the same items/elements across
project years one and two. Based on these and other concerns, we will be obtaining
assistance from the U.S. Department of Education’s Data Quality Initiative contractor to
create and disseminate detailed data collection and reporting guidance for the program,
as well as standards that we will use to determine what constitutes valid data for this
measure. After that project is complete in 2008, we will aggregate data based on these
standards and report it.

FY 2008 Performance Targets. We established targets for percentage of grantees
experiencing a 5 percent reduction in past-year illegal drug use after reviewing the first
two years of data for the FY 2003 cohort of grant sites. Consistent with research that
suggests that changes in student behavior related to student drug testing may not be
realized immediately, we assumed that we could look for an increased number of
grantees to experience positive change and, using our professional judgment, set that
target at 50 percent of grantees. When we have received data for three project years
from a single cohort of sites, we’ll revisit targets for future cohorts. We may also need
to consider revising targets for future cohorts based on full results from the FY 2005
cohort because the FY 2003 cohort is very small (only 8 sites) and may not be typical of
other cohorts.

Methodology Data are collected by grantees using student surveys. Data are provided
as part of the grantees’ annual performance reports. Grantees do not use the same
survey items to collect data for this measure but, rather, self-select survey items (often
from surveys already administered) in order to provide this data. Survey items may
relate to different substances, but must collect information concerning annual use in
order to be included in the data reported for this measure.

Authorized representatives for the grant site sign the annual performance report and, in
doing so, certify that to the best of the signer’s knowledge and belief, all data in the
performance report are true and correct and that the report fully discloses all known
weaknesses concerning the accuracy, reliability, and completeness of the data
included. Generally, the Department relies on the certification concerning data supplied
by grantees and does not conduct further reviews.




                                            6

The levels of anticipated decrease in substance abuse are consistent with those
included in the National Drug Control Strategy – five percent per year. Targets were
established following the report of baseline data for grant sites from the FY 2003 cohort.
When we have received data for three project years from a single cohort of sites, we’ll
revisit targets for future years in other cohorts.

              Safe and Drug-Free Schools and Communities State Grants

Measure 4: The percentage of students in grades 9-12 who were offered, sold, or
given an illegal drug on school property during the past 12 months (Safe and Drug-Free
Schools State Grants)

Table 4

 FY 2003    FY 2004      FY 2005    FY 2006       FY 2007    FY 2007     FY 2008
  Actual     Actual       Actual     Actual        Target     Actual      Target
29          None        25          None         27          Pending     None

The measure. This measure is one of three measures directly related to reducing
student drug or alcohol use for the Safe and Drug-Free Schools and Communities
(SDFSC) State Grants. This formula grant program provides funds to the States, based
on school-aged population and the State’s relative share of Elementary and Secondary
Education Act Title I concentration grant funds, to support drug and violence prevention
programs. The measure directly relates to the National Drug Control Strategy Goal of
preventing youth drug use by focusing on the extent to which illegal drugs are available
on school property.

FY 2007 Performance Results. FY 2007 data will be available in September 2008. We
established targets for this measure by reviewing trends in this measure over time and
identifying a target that reflects continuing achievable reductions.

FY 2008 Performance Targets. There are no targets for this program in FY 2008
because data are collected only in the odd-numbered years.

Methodology. Data for this measure are collected from a nationally representative
sample of students in grades 9-12 as part of the Youth Risk Behavior Surveillance
System (YRBSS), sponsored by the Centers for Disease Control and Prevention (CDC).
Data are collected in odd years and reported in the following even years. No data are
collected for even years and, as a result, no targets have been established for even
years.

Detailed information about the methodology used to sample and report data for the
YRBSS is available at the CDC website at:
http://www.cdc.gov/mmwr/preview/mmwrhtml/ss5505a1.htm. We rely on the assertions
provided about methodology presented by CDC in using this data to report on
performance of SDFSC State Grants.


                                            7

Measure 5: The percentage of students in grades 9-12 who used marijuana one or
more times during the past 30 days. (SDFSC State Grants)

Table 5

 FY 2003   FY 2004      FY 2005    FY 2006       FY 2007   FY 2007     FY 2008
  Actual    Actual       Actual     Actual        Target    Actual      Target
22         None        20          None         19         Pending     None

The measure. This measure is one of three measures directly related to reducing
student drug and alcohol use for SDFSC State Grants. This formula grant program
provides funds to the States, based on school-aged population and the State’s relative
share of Elementary and Secondary Education Act Title I concentration grant funds, to
support drug and violence prevention programs. The measure is directly related to the
National Drug Control Strategy Goal of preventing youth drug use by focusing on the
extent of current use by high school aged-youth of the most prevalent illegal drug.

FY 2007 Performance Results. FY 2007 data will be available in September 2008.

FY 2008 Performance Targets. There are no targets for this program in FY 2008
because data are collected only in the odd-numbered years.

Methodology. Data for this measure are collected from a nationally representative
sample of students in grades 9-12 as part of the Youth Risk Behavior Surveillance
System (YRBSS), sponsored by the Centers for Disease Control and Prevention (CDC).
Data are collected in odd years and reported in the following even years. No data are
collected for even years and, as a result, no targets have been established for even
years.

Detailed information about the methodology used to sample and report data for the
YRBSS is available at the CDC website at:
http://www.cdc.gov/mmwr/preview/mmwrhtml/ss5505a1.htm. We rely on the assertions
provided about the methodology presented by CDC in using this data to report on
performance of SDFSC State Grants.

Measure 6: The percentage of students in grades 9-12 who had five or more drinks of
alcohol in a row one or more times during the past 30 days. (SDFSC State Grants)

Table 6

 FY 2003   FY 2004      FY 2005    FY 2006       FY 2007   FY 2007     FY 2008
  Actual    Actual       Actual     Actual        Target    Actual      Target
28         None        26          None         26         Pending     None




                                           8

The measure. This measure is one of three measures related to reducing student drug
or alcohol use for SDFSC Grants. This formula grant program provides funds to the
States, based on school-aged population and the State’s relative share of Elementary
and Secondary Education Act Title I concentration grant funds, to support drug and
violence prevention programs. The measure is directly related to the National Drug
Control Strategy Goal of preventing youth drug use by focusing on the prevalence of
binge drinking by high school aged-students. While alcohol is not explicitly an emphasis
of the National Drug Control Strategy, illegal use of alcohol can be associated with use
of other illegal drugs.

FY 2007 Performance Results. FY 2007 data will be available in September 2008.

FY 2008 Performance Targets. There are no targets for this program in FY 2008
because data are collected only in the odd-numbered years.

Methodology. Data for this measure are collected from a nationally representative
sample of students in grades 9-12 as part of the Youth Risk Behavior Surveillance
System (YRBSS), sponsored by the Centers for Disease Control and Prevention (CDC).
Data are collected in odd years and reported in the following even years. No data are
collected for even years and as a result no targets have been established for even
years.

Detailed information about the methodology used to sample and report data for the
YRBSS is available at the CDC website at:
http://www.cdc.gov/mmwr/preview/mmwrhtml/ss5505a1.htm. We rely on the assertions
provided about the methodology presented by CDC in using this data to report on
performance of SDFSC State Grants.

Measure 7: The percentage of drug and violence prevention programs/practices
supported with SDFSC State Grant funds that are research based. (SDFSC State
Grants)

Table 7

FY 2003    FY 2004      FY 2005    FY 2006      FY 2007    FY 2007      FY 2008
 Actual     Actual       Actual     Actual       Target     Actual       Target
None       None        7.8         None         None       None        11.7

The measure. This measure examines the extent to which programs and practices
supported with SDFSC State Grant funds are based on research. The measure
supports attainment of National Drug Control Strategy goals by focusing on the quality
of programs supported with SDFSC State Grants funds and the likelihood that the
programs will reduce or prevent youth drug use.

FY 2007 Performance Results. No target is established for this measure in FY 2007;
data are collected every three years.


                                           9

FY 2008 Performance Targets. A target of 11.7 percent has been established for FY
2008. We considered the challenges involved in encouraging the adoption of research-
based programs and, using our professional judgment, we established a target for a 50
percent increase over the baseline in the prevalence of research-based programs
implemented in schools. Although the target appears modest, it requires change in a
very significant number of schools and districts in order to be achieved.

The next data collection is scheduled to collect information about programs
implemented during the current (2007-2008) school year, so our remaining ability to
influence performance on this measure for the next target is limited. However, we have
established this issue as one of our strategic priorities for the Office of Safe and Drug-
Free Schools 2008 organizational assessment. As part of that effort, SDFSC State
Grant program staff have developed a plan to emphasize adoption and implementation
of research-based programs. Strategies include requesting State protocols/standards
for approving applications and providing waivers, focusing phone monitoring contacts
on this issue, examining protocols to identify compliance issues and technical
assistance needs, and identifying and disseminating best practices in this area based
on protocols and monitoring activities.

Methodology. Baseline data for this measure were collected from a nationally
representative sample of schools under a contract supported by ED. As a first step, the
contractor developed a large list of research-based programs and then screened those
programs to identify programs that were relevant to the SDFSC State Grants program;
had at least two empirical studies completed that met stringent methodological
standards; had implementation materials available; used at least two independent
samples in program evaluations; and demonstrated an adequate level of program
effectiveness.

The contractor collected data for the measure using surveys of national probability
samples of public elementary and secondary schools and the school districts with which
they were associated. The surveys – conducted using both mail and web-based
approaches – gathered information on prevention programs operating during the 2004-
2005 school year. Survey information was collected between fall 2005 and spring 2006.

The sample design included 2,500 districts, and nearly 6,000 schools that were
sampled from the 2,500 districts. The contractor used the National Assessment of
Educational Progress (NAEP) national sample frame. The NAEP sample frame is
derived from the 2003-2004 National Center for Education Statistics Common Core of
Data (CCD) Public Elementary and Secondary School Universe and Agency files.
Using the NAEP sample frame allowed the contractor to take advantage of edits already
made to the CCD files (for example eliminating administrative school districts from the
sample frame).




                                            10

Survey instruments used included 89 prevention programs; respondents were also able
to write in any programs omitted from those listed. The contractor received responses
from 91 percent of the districts included in the sample and 86 percent of schools.
The study conducted by the contractor to obtain data for this measure has some
limitations that are the result of both the research synthesis and survey data collections.
Despite significant efforts to be comprehensive, it is possible that the literature searches
used may not have identified some published studies on prevention programs, and as a
result, the number of research-based program may be understated.

Some other study limitations pertain to the quality of data collected via the surveys.
Recall problems and responses from less knowledgeable respondents in some schools
and districts (particularly among schools and districts that provided information late in
the collection period) may have affected the quality of data. Schools may have also
over-reported the prevention programs operating in their schools because respondents
confused the specific named program in the survey with other similarly named but
different programs.

Data for this measure will be collected every three years. A second data point for this
measure will be collected in 2008-2009; information will reflect 2007-2008 school year
practices.

Measure 8: The percentage of drug and violence prevention curriculum programs that
are implemented with fidelity. (SDFSC State Grants)

Table 8

FY 2003     FY 2004      FY 2005     FY 2006       FY 2007    FY 2007      FY 2008
 Actual      Actual       Actual      Actual        Target     Actual       Target
None        None        44.3         None          None       None        50.9

The measure. This measure examines the extent to which research-based curriculum
programs supported with SDFSC State Grant funds are implemented with fidelity. The
measure supports attainment of National Drug Control Strategy goals by focusing on
the quality of implementation of the research-based programs and practices supported
with SDFSC State Grants funds, and the corresponding likelihood that the programs will
reduce or prevent youth drug use.

FY 2007 Performance Results. No targets were established for this measure for FY
2007 since the data are collected every three years.

FY 2008 Performance Targets. We established a target of 50.9 percent for FY 2008.
We considered the challenges involved in encouraging the adoption of research-based
programs and, using our professional judgment, we established a target for a 15
percent increase over the baseline data in the percentage of drug prevention programs
that are implemented with fidelity. Although the target appears modest, it means
change in a very significant number of schools and districts in order to be achieved.


                                             11

Methodology. Baseline data for this measure were collected from a nationally
representative sample of schools under a contract supported by ED. Data were
collected in the fall of 2006, and reflected information about programs and practices
implemented during the 2004-2005 school year. The contractor developed a list of
research-based programs and compared information about programs and practices
being implemented with SDFSC State Grants funds with the list of research-based
program and practices. (See discussion for Measure 7)

The contractor then followed up with a subset of respondents to examine the extent to
which research-based programs and practices were implemented in a manner
consistent with implementation keys for individual programs (as determined by program
developers). The contractor focused its review on the 10 programs (from the list of 21
research-based programs) that were implemented most frequently by respondents to
the initial phase of the study.

The contractor mailed copies of questionnaires to principals and program implementers
to each school that reported operating at least one research-based program in the
response to the earlier survey. The response rate for the questionnaire supplied to
program implementers was 78 percent; the response rate for questionnaires completed
by principals was 70 percent.

The study developed program-specific measures of quality implementation for each of
the research-based programs identified by the study. The standards were based on
program developer’s specifications for individual programs. Aspects of implementation
considered included issues such as frequency of student participation; number of
lessons delivered; and topics covered. Based on applying these quality standards to
data supplied on the two questionnaires, the contractor identified the percentage of
research-based programs that were implemented according to the standards identified
by the program developer (which the study refers to as being implemented with
“fidelity”).

This aspect of the study has some limitations related to the application of the program-
specific standards for assessing the quality of program implementation and from the
additional survey data questionnaires. Valid measurement of quality of implementation
required that a program developer’s program specifications be applied to implementer
reports on that specific program. In some cases, responses raised questions about
whether respondents were reporting on the correct program. Study staff worked to
confirm that implementers were reporting on the correct program; in cases where the
implementer reported on the wrong program, that report was considered invalid and not
included in the final data. If responses suggested that the program implementer
reported on the wrong program and confirmation could not be made, those cases were
also excluded from analyses.

Similar problems occurred for programs that had multiple components or different
versions that are implemented for different ages or grade levels. Study staff reviewed



                                           12

program materials for different components or versions and worked to identify the
program standards most closely related to the various components or versions. If a
meaningful standard for a component or measure could be developed, the case was
included in the analyses; if not, the program was omitted.

Limitations related to data quality from questionnaires also exist. Because a substantial
number of cases were ineligible for inclusion in the study analyses for the reasons
described above, the number of valid cases was reduced, leading in turn to decreased
precision in estimates and larger than expected standard errors and confidence
intervals. Similar recall problems caused by the gap between program implementation
and data collection (as discussed for the previous measure) may have also impacted
data quality. Finally, the quality of reports varied by the extent to which respondents
were in a position to observe actual implementation and intentionally bias reports.
Program implementers may have difficulty in providing objective information about
programs they are responsible for establishing. However, previous research using
similar measures suggests that this ‘social desirability’ bias is likely to be low.

Data for this measure will be collected every three years. A second data point for this
measure will be collected in 2008-2009; information will reflect 2007-2008 school year
practices.

                            Grants to Reduce Alcohol Abuse

Measure 9: The percentage of grantees whose target students show a measurable
decrease in binge drinking (Grants to Reduce Alcohol Abuse Program – FY 2004 and
FY 2005 cohorts)

Table 9

Cohort    FY         FY         FY        FY         FY         FY         FY
          2003       2004       2005      2006       2007       2007       2008
          Actual     Actual     Actual    Actual     Target     Actual     Target
2004      n/a        n/a        n/a       50         70         pending    n/a
2005      n/a        n/a        n/a       n/a        none       65.2%      75

The measure. This measure examines a key outcome for the Grants to Reduce Alcohol
Abuse (GRAA) program – reduction in binge drinking for the target population. While
the National Drug Control Strategy is focused most intensively on preventing the use of
controlled substances, the strategy does address the role of alcohol as a drug of choice
for teenagers. Data do suggest that early use of alcohol is more likely to result in heavy
later use of alcohol.

FY 2007 Performance Results. Actual performance data for FY 2007 for the FY 2004
cohort will be contained in final reports for these grant projects. Grants are currently
operating in no-cost extensions; generally, final reports will be due at the end of 2008
for this cohort. Data will be aggregated and available in March 2009.


                                            13

No FY 2007 target exists for the FY 2005 cohort for this measure since the first two
performance reports were necessary in order to establish baseline data for this cohort.

FY 2008 Performance Targets. We have established a performance target of 75
percent for FY 2008 for this measure for the FY 2005 cohort. We elected to adjust the
target for this measure to reflect the fact that a significantly greater number of grantees
posted decreases in binge drinking at baseline. The target of 75 percent reflects an
increase over the year three target for the prior (FY 2004) cohort, but a smaller increase
between baseline and year three because of the performance of the FY 2005 cohort of
grantees at baseline.

Methodology. Data for this measure are collected by grantees and reported as part of
annual performance reports. If data for this measure are not available at the time that
performance reports are submitted, staff follow up with sites to attempt to obtain data for
the measure. Grantees that fail to provide data are not included in the tabulation of data
for the measures. Also, grantees that did not provide data for two consecutive project
years (so that we could determine if a decrease in binge drinking had occurred) are not
included in data report for the measure. Authorized representatives for the grant site
sign the annual performance report and, in doing so, certify that to the best of the
signer’s knowledge and belief, all data in the performance report are true and correct
and that the report fully discloses all known weaknesses concerning the accuracy,
reliability, and completeness of the data included. Generally, the Department relies on
the certification concerning data supplied by grantees and does not conduct further
reviews.

ED does not mandate data collection protocols or instruments for grantees. Grantees
select a survey item that reflects the concept of binge drinking, and collect and report
data about that survey item as part of their performance reports. As a result, data are
not comparable across grant sites, but individual grant sites are required to use the
same survey items across performance periods.

Applicants are not required to furnish baseline data as part of their applications. Data
supplied after year one are considered baseline data for the projects. Projects require
two years of data in order to determine if a decrease in binge drinking among target
students has occurred.

Measure 10: The percentage of grantees that show a measurable increase in the
percentage of target students who believe that alcohol abuse is harmful to their health
(Grants to Reduce Alcohol Abuse – FY 2004 and FY 2005 cohorts)




                                            14

Table 10

Cohort     FY        FY         FY         FY         FY         FY         FY
           2003      2004       2005       2006       2007       2007       2008
           Actual    Actual     Actual     Actual     Target     Actual     Target
2004       n/a       n/a        n/a        55.6       76         pending    n/a
2005       n/a       n/a        n/a        n/a        none       70.1       80

The measure. This measure examines a key outcome for the Grants to Reduce Alcohol
Abuse (GRAA) program – perception of health risk for alcohol abuse among target
students. While the National Drug Control Strategy is focused most intensively on
preventing the use of controlled substances, the Strategy does address the role of
alcohol use as a drug of choice for teenagers. Data do suggest that changes in
perceptions about risks to health resulting from alcohol use are positively correlated with
reductions in alcohol use.

FY 2007 Performance Results. Actual performance data for FY 2007 for the FY 2004
cohort will be contained in final reports for these grant projects. Grants are currently
operating in no-cost extensions; generally, final reports will be due at the end of 2008
for this cohort. Data will be aggregated and available in March 2009.

No FY 2007 target exists for the FY 2005 cohort for this measure since the first two
performance reports were necessary in order to establish baseline data for this cohort.

FY 2008 Performance Targets. We have established a performance target of 80
percent for FY 2008 for this measure for the FY 2005 cohort. We elected to adjust the
target for this measure to reflect the fact that a significantly greater number of grantees
posted decreases in binge drinking at baseline than was the case for the FY 2004
cohort. The target of 80 percent reflects an increase over the year three target for the
prior (FY 2004) cohort, but a smaller increase between baseline and year three
because of the performance of the FY 2005 cohort of grantees at baseline.

Methodology. Data for this measure are collected by grantees and reported as part of
annual performance reports. If data for this measure are not available at the time that
performance reports are submitted, staff follow up with sites to attempt to obtain data for
the measure. Grantees that fail to provide data are not included in the tabulation of data
for the measures. Also, grantees that did not provide data for two consecutive project
years (so that we could determine if an increase in the percentage of students that
believe that alcohol abuse is harmful to their health had occurred) are not included in
data report for the measure. Authorized representatives for the grant site sign the
annual performance report and, in doing so, certify that to the best of the signers’
knowledge and belief, all data in the performance report are true and correct and that
the report fully discloses all known weaknesses concerning the accuracy, reliability, and
completeness of the data included. Generally, the Department relies on the certification
concerning data supplied by grantees and does not conduct further reviews.


                                             15

ED does not mandate data collection protocols or instruments for grantees. Grantees
select a survey item that reflects the concept of binge drinking, and collect and report
data about that survey item as part of performance reports. As a result, data are not
comparable across grant sites, but individual grant sites are required to use the same
survey items across performance periods.

Applicants are not required to furnish baseline data as part of their applications. Data
supplied after year one are considered baseline data for the projects. Projects require
two years of data in order to determine if a decrease in binge drinking among target
students has occurred.

Measure 11: The percentage of grantees that show a measurable increase in the
percentage of target students who disapprove of alcohol abuse (Grants to Reduce
Alcohol Abuse – FY 2004 and FY 2005 Cohorts)

Table 11

Cohort     FY        FY         FY        FY         FY         FY         FY
           2003      2004       2005      2006       2007       2007       2008
           Actual    Actual     Actual    Actual     Target     Actual     Target
2004       n/a       n/a        n/a       66.7       87         pending    n/a
2005       n/a       n/a        n/a       n/a        none       71.2       87

The measure. This measure examines a key outcome for the Grants to Reduce Alcohol
Abuse (GRAA) program – perception of health risk for alcohol abuse among target
students. While the National Drug Control Strategy is focused most intensively on the
preventing the use of controlled substances, the Strategy does address the role of
alcohol use as a drug of choice for teenagers. Data do suggest that increases in the
percentage of target students who believe that alcohol abuse is not socially acceptable
are associated with declines in consumption of alcohol.

FY 2007 Performance Results. Actual performance data for FY 2007 for the FY 2004
cohort will be contained in final reports for these grant projects. Grants are currently
operating in no-cost extensions; generally. Final reports will be due at the end of 2008
for this cohort. Data will be aggregated and available in March 2009.
No FY 2007 target exists for the FY 2005 cohort for this measure since the first two
performance reports were necessary in order to establish baseline data for this cohort.

FY 2008 Performance Targets. We have established a performance target of 87
percent for FY 2008 for this measure for the FY 2005 cohort. We retained the target
from the FY 2004 cohort for this measure since it is an aggressive target (requiring that
nearly all grantees achieve improvement for this measure), and the baseline differences
between cohorts were smaller than in the other two measures for this program.

Methodology. Data for this measure are collected by grantees and reported as part of
annual performance reports. If data for this measure are not available at the time that


                                            16

performance reports are submitted, staff follow up with sites to attempt to obtain data for
the measure. Grantees that fail to provide data are not included in the tabulation of data
for the measures. Also, grantees that did not provide data for two consecutive project
years (so that we could determine if an increase in the percentage of students that
disapprove of alcohol abuse had occurred) are not included in data report for the
measure. Authorized representatives for the grant site sign the annual performance
report, and in doing so, certify that to the best of the signer’s knowledge and belief, all
data in the performance report are true and correct and that the report fully discloses all
known weaknesses concerning the accuracy, reliability, and completeness of the data
included. Generally, the Department relies on the certification concerning data supplied
by grantees and does not conduct further reviews.

ED does not mandate data collection protocols or instruments for grantees. Grantees
select a survey item that reflects the concept of binge drinking, and collect and report
data about that survey item as part of performance reports. As a result, data are not
comparable across grant sites, but individual grant sites are required to use the same
survey items across performance periods.

Applicants are not required to furnish baseline data as part of their applications. Data
supplied after year one are considered baseline data for the projects. Projects require
two years of data in order to determine if a decrease in binge drinking among target
students has occurred.

Assertions
                             Performance Reporting System

The Department of Education has a system in place to capture performance information
accurately and that system was properly applied to generate the performance data in
this report. In instances in which data are supplied by grantees as part of required
periodic performance reports, the data that are supplied are accurately reflected in this
report.

Data related to the drug control programs included in this Performance Summary
Report for Fiscal Year 2007 are recorded in the Department of Education’s software for
recording performance data and are an integral part of our budget and management
processes.
                   Methodology for Establishing Performance Targets

The methodology described in the Performance Summary Report for Fiscal Year 2007
to establish performance targets for the current year is reasonable given past
performance and available resources.




                                            17

             Performance Measures for Significant Drug Control Activities

The Department of Education has established at least one acceptable performance
measure for each Drug Control Decision Unit identified in its Detailed Accounting of
Fiscal Year 2007 Drug Control Funds.

Criteria for Assertions

                                          Data

No workload or participant data support the assertions provided in this report. Sources
of quantitative data used in the report are well documented. These data are the most
recently available and are identified by the year in which the data was collected.

                               Other Estimation Methods

No estimation methods other than professional judgment were used to make the
required assertions. When professional judgment was used, the objectivity and strength
of those judgments were explained and documented. Professional judgment was used
to establish targets for programs until data from at least one grant cohort were available
to provide additional information needed to set more accurate targets. We routinely re-
evaluate targets set using professional judgment as additional information about actual
performance on measures becomes available.

                                   Reporting Systems

Reporting systems that support the above assertions are current, reliable, and an
integral part of the Department of Education’s budget and management processes.
Data collected and reported for the measures discussed in this report are stored in the
Department of Education’s Visual Performance System (VPS). The VPS includes
appropriate disclosures about data quality issues associated with measures. Data from
the VPS are used in developing annual budget requests and justifications, and in
preparing reports required under the Government Performance and Results Act of
1993.




                                           18