GAO's Review of USDA's National WIC Evaluation Report and Follow-up Issues

Published by the Government Accountability Office on 1990-01-24.

Below is a raw (and likely hideous) rendition of the original report. (PDF)

                     United    States General        Accounting      Office

For Release   on     GAO's         Review of USDA's National                  WIC Evaluation
Delivery             Report         and Follow-up  Issues
Expected   at
9:00 a.m. EDT
January   24, 1990

                     Statement          of Keith  0. Fultz
                     Director         of Planning   and Reporting

                     Resources,            Community,          and   Economic
                     Development            Division

                     Before     the House Select       Committee       on Hunger;
                     the Senate      Committee      on Agriculture,        Nutrition,
                     and Forestry;      and the Subcommittee            on
                     Nutrition,      and Investigations,           Senate Committee
                     on Agriculture,       Nutrition,       and Forestry

                          \ dqf;      3;      ’ , c..&)ij&-q
                                                                                    GAO   Form160<12/87)
          Messrs.              Chairmen           and members              of         the        Committees:

          We are           pleased           to    discuss          our         report            on the           Department             of
Agriculture's                   (USDA)        handling             of     the         National             WICl         Evaluation,                 as

well      as its           research           on     the       program's                   impact       on children.*                     This

report          was reguested                 by     the       Chairman               of      the      Select           Committee              on
Hunger,          Senators              Tom Harkin              and James                   Jeffords,              and     Representative

Augustus           Hawkins.

           In    summary,              USDA deleted                the      original                 chapter             and     executive

summaries            of        the     National            WIC Evaluation                         and replaced                  them    with          its

compendium                of     results.            We believe                 that          the      original               executive

summary          used          appropriate               methodology,                      was      accurately                presented,                 and

pointed          out       the        positive           effects          attributable                       to     the       WIC Program.

In     contrast,               USDA's        compendium                 contained                 errors           and misleading

statements                about        some of           the    data        and            deleted           the        study       team's

overall          conclusions                 regarding             the      WIC Program's                          impact        on

participants.                        We also        found        that       a combination                          of     factors       caused

the       National             WIC Evaluation                  report            to        take      3-l/2          years        longer             and

cost       $2 million                 more        than     estimated              to         complete.                  For     example,

 'Special            Supplemental                  Food        Program           for         Women,          Infants,            and

 *Our report    entitled     Food Assistance:                                             The          National  WIC
 Evaluation:     Reporting       and Follow-up                                         Issues           (GAO/RCED-90-3,                      Dec.
 14, 1989)   is being     publicly    released                                         today.
about       2 years            into          the        study,              USDA replaced                         the        study's               principal

investigator               and redesigned                             the        study.

          In     1983,         USDA proposed                          another                    study--estimated                         to        cost        about

$2 million--              to     determine                    WIG's          effects                   on children                  of        the

participants               in        the          National                 WIC Evaluation.                              USDA withdrew                       the

proposal           prematurely                     in     1984.               In           1987        USDA initiated                          efforts             to

determine            the        feasibility                      of        conducting                    a more             comprehensive                       child

impact          study          on an entirely                          new sample                       population.                      In        1989      USDA

staff          estimated              that          the         study         proposed                   by its             contractor                   will           cost

$16-22          million           and         take         5-l/2            to        6-l/2             years          to     complete.                    USDA

currently            has        a contract                      to     address                   its     concerns                 about            the

technical            feasibility                        and proposed                         costs         of      the        new child                   impact
 study.           We believe                  that         it         is     important                    for      USDA, when                      conducting

major          studies          of     WIG's              impact,                to         learn         from         the        mistakes                made          in

 the      National             WIC Evaluation.

           The balance                  of        my      testimony                       will         provide              you    with            a brief
background                on the           National                   WIC Evaluation                            and address                    the         flaws             in

 USDA's          compendium                  of     results,                 the            factors             that         delayed               release              of

 the      National             WIC Evaluation,                             the            premature              cancellation                       of      the

 follow-up           study            on WIC children,                                    the      current             efforts                to     assess

 WIG's         impact          on child                 development,                             and    some       factors               to        consider                  in
 future          evaluations                  of        the          WIC Program.


           The     WIC Program                       is      a federally                    funded          nutrition                   assistance

program           that         provides               supplemental                         foods,         nutrition                education,                     and

referral           to       health             care          to     low-income                     pregnant,                breast-feeding,

postpartum                women:          infants:                  and children                        up to         5 years            old.

Although           USDA has               undertaken                      many            studies          of        specific            aspects             of

the     WIC Program,                     the         National              WIC Evaluation,                            which        began          in     1979

and     was       released               in      1986,             was     the           most       complex             and      comprehensive

evaluation                of     the      program.                       This         congressionally                        mandated             study            was

USDA's         second            attempt               to        generate                 useful          information                   on the

effectiveness                    of      the         WIC Program.

           The      first             attempt--              commonly                known          as the            Endozien             study,            after

its     principal                investigator--was,                                      according              to    USDA and others,

severely            criticized                   by researchers                            working              in     the      field       because                of

its     research                methodology.                         As a consequence,                               that       study's           results

generally                lacked          credibility                      with            the      scientific                community.                  To

avoid        the         credibility                      problems              of        the      Endozien             study,           USDA and its

National            WIC Evaluation                               study      team            had      to    carefully               design              and

conduct            the         research               to     produce                 a credible                 study         which        would
withstand                the      scrutiny                  of     the      scientific                    community.

            The     end         result           of         the      National                   WIC Evaluation                   was       a report
which         established                     WIG's          effectiveness                         in     several             areas        including:

 (1)       improving              the         diet          of     pregnant                 women         and children,                     (2)

lengthening              gestation                  and      reducing              the     likelihood                      of     pre-term

deliveries,               and         (3)        increasing             weight            gain          and the             use of              prenatal

care       in    pregnant             mothers.                   However,           almost              immediately                     after          the

study's          release,              the        principal             investigator                      for        the         study          team         and

Members          of     Congress                 raised          concerns           regarding                   the        manner          in      which

the     study          was reviewed,                      revised,            and released                      by     USDA.



           One of         the        areas          that         concerned               the         principal                  investigator

and Members               of      Congress                was the         compendium                    of      results             which          was

written          by     USDA and                 substituted                for     the         executive                  and      chapter
summaries              written              by    the      research               team.              We compared                   USDA's

compendium              of      results             and      the       research                team's           executive                 summary

with       the        5-volume              National             WIC Evaluation                       report.                   While      we found

only       one minor              error            in     the      research              team's              summary,              we found

substantial                  flaws          in    the      compendium               which             serve           to        understate--and
mislead          the      reader             about--            the    generally                positive               effects             of      the        WIC


           Specifically,                     USDA's             synthesis           of         the      National                 WIC

Evaluation's                   results             included            inappropriate                     methodological                           steps

and       reporting              inaccuracies.                        There        are         four          areas         that         could          serve

to     understate               the         benefits             of    WIC participation                              and mislead                  the

          First,          USDA did              not       preserve              the        original                 research              design              used

by    the     team.             The         National            WIC Evaluation                     was designed                         in        a

hierarchical                fashion,              with          6 major             research             issues                 divided               into        15

subissues.                 Conclusions                  regarding               the        subissues                    were       made by

examining             certain               indicators             (or     measures)                    of     those             subissues.

USDA's        compendium                    eliminated            this         hierarchy                 and             used      categories
created            by USDA to                report         results.                  Some        of     these                categories                 used

National            WIC Evaluation                      subissues,                  others         combined                     two     subissues                      or
created            new ones.                 Thus       the      compendium                  represented                        USDA's            analysis

of    the     National                WIC Evaluation                     rather            than         a summary                  of     the

evaluation                team's            findings.              This         difference                    was not                 reported               to

the    reader,             consequently                   the      reader             is     misled                to     assume             that        the

compendium                was a summary                   of     the       findings               of         the         National             WIC

            Second,         USDA's             compendium                provided              summary                   statistics                   that
were        not      accurate.                 For       example,              USDA condensed                            results             of       over            100

measurements                related              to      diet      and         reported                that             there         were        only            5

indications                of        diet      assessed,               thus         generally                 understating                        WIG's
impact         on program                   participants.

            Third,         USDA incorrectly                        reported                conflicts                     in     the
significance                    of     outcomes            reported             by the            National                    WIC Evaluation.

For     example,                USDA's         compendium                reports             a conflict                       between             the
 findings            of    the         Historical               Study          of     Pregnancy                    Outcomes               and         the

Longitudinal                     Study           of      Pregnant                 Women--two                       of      the            four      component

studies            of      the         National                WIC Evaluation--regarding                                                   WIG's           impact            on

late       fetal           death.                We found                  that        there                were         no conflicts                        in        the

results            of       these            studies,                merely            a difference                            in     the          statistical

significance                     of     the           effects              reported.                         The compendium's                              treatment

of     this          issue            leads           the      reader             to        believe                 that            the        Evaluation                    was

unable           to       determine                   the      WIC Program's                            impact             on late                 fetal           death.

The       National               WIC Evaluation                            actually                    found            a significant                        reduction

of     2.3       deaths               per        1,000          attributable                           to      the       WIC program.                             In

essence              by       not      reporting                     the     conclusions                           reached                by     the       research

team,          the        compendium                    deprives                 the        lay         reader             of        access             to        the

bottom-line                    findings                 of      the        National                    WIC Evaluation,                             and thus

misleads                the         reader.

              Finally,                USDA incorrectly                             reported                    the       Evaluation's                        finding               on

health           services                   by    omitting                 certain                    indicators.                         For      example,                  USDA

did       not         report           the        significant                      impact                   that        WIC benefits                       had          on

attaining                  a "regular                   source             of     medical                    care"         for            three         age groups.
 Such         omissions                may lead                 a reader                   to         assume            that          the         researchers

 found          a much weaker                         WIC effect                   than           was          actually                   the      case.


 OF THE NATIONAL                            WIC EVALUATION

              Although                USDA estimated                            that            the         National                WIC Evaluation

 would          be        completed               in         2-l/2         years                and         cost        about              $3.9        million,               the

study        took        about              6 years              and      cost           $5.9       million.                   Release           of        the

study        was delayed                      for       four          principal                  reasons:                (1)        USDA officials

replaced             the        study's                principal                   investigator                  and       the       new

investigator                    redesigned                    the       study,              (2)      the     research                team         had

difficulty                 producing                   a product                   acceptable              to         USDA officials                       and        the

study's            advisory                 panel           within            the         time       frames            estimated,                 (3)       USDA's

review          was protracted                          because               it      wrote          a compendium                    of      results               to

replace            the         study          team's             executive                 summary,              and         (4)     unforseen

printing             problems                 were          encountered.

             The      National                WIC Evaluation                          began          in    September                  1979 with                  award

of      a contract                   to     the        Research               Triangle               Institute.                     To avoid               the

acceptance                 problems                 USDA encountered                              with     the         Endozien              study,              USDA

established                    and         used        a panel            to        advise           it    on         study         design,             scope,

and      report             presentation.                           The       panel,              consisting               of       outside             experts,

included              expertise                   in    WIC administration,                                medical                 research,               and

research              methodology.                          After         reviewing                  the     design,                which         was

submitted                for         approval                 about          2 years              after         the      study         began,              the

advisory              panel           recommended                      that         USDA redesign                      the         study         and       place

it      under         the       direction                   of      a medical                researcher.                       As a result,                      USDA
selected              Dr.         David           Rush         as principal                       investigator.                       Dr.        Rush

redesigned                  the           study        in      9 months                  and began              his      field            work        in     the

 summer         of       1982.              Most         of      the      data            were       collected                 by the            end       of

1983,         although                    analysis             of      the         data          continued             into         late         1984.             The
first         draft            of     the         National              WIC Evaluation,                          submitted                  in    May 1984

 to     USDA and               its         advisory              panel,              was poorly                 written,              according                  to

USDA,       because             the           study            team          was attempting                              to         analyze                the      data        and

draft       reports             on        four           component                     studies                  simultaneously.                               USDA

changed           the        format               of     the         report             and         gave           the         research                    team

additional               time           to        prepare              a final                product.

          The        final          draft               of     the       National                   WIC Evaluation                               report,

satisfying               all        of        USDA staff's                         technical                     comments,                    was delivered                       to

USDA in           February                1985.                The       Administrator                             of     the             Food         and Nutrition

Service           and        the         Assistant                    Secretary                    for          Food          and Consumer                        Services
normally             take          about               8 weeks           to        review                 USDA reports                          for        policy

implications.                       However,                   the       National                   WIC Evaluation                               took            9 months

to      review.              The         review               was protracted                              because                  the      chapter               and

executive               summaries                      written           by        the        research                   team             were         deleted             in

the      Office          of        the        Assistant                  Secretary                        for         Food          and Consumer

Services             and were                 replaced                 with            the         compendium                       of      results,'which
USDA staff               wrote.                   USDA told                   us that                it         replaced                  the         research

team's          summaries                    with            the      compendium                    because                   it      believed                   that      the

 summaries              misstated                      the         results             of     the          National                      WIC Evaluation.

           Finally,                in        attempting                  to        expedite                     the      printing                     of     the        report,
USDA misinterpreted                                    the         Government                     Printing               and             Binding

 Regulations.                      This            resulted                  in    the        Department                           printing                a different

 number         of      copies               of        each         volume             of     the           report                 and      only           50 copies              of

 the     entire          report.                       The         quantities                 printed                   were             insufficient,                     given

 the     amount          of        public                interest                 in        the      study.                   Release                 of     the        final

                                                                                       8                                                                     /
report       was         further             delayed              because              the        initial                  printing               was

unreadable,               and a second                     printing               was required.


           While         the         National             WIC Evaluation                           was         in        progress,                researchers

found       indications                     that         WIC improved                   head             size            and       perhaps             brain

growth        and behavioral                          and cognitive                     development                          in     children.                      These

issues        were        outside               the       scope       of         the         original                     study.            However,                USDA
was      convinced              that          these         potential                  beneficial                         impacts           on children

merited           further             examination                   and     therefore                      planned                 to     conduct              a

follow-up               study         of      the        children           born             to     mothers                 who participated                          in
the      National              WIC Evaluation.                            USDA solicited                                 such      a study             in      1983

but      withdrew              the         request          in      1984         citing              its            concern              that,         because

of    a confidentiality                             pledge          given         to         National                     WIC Evaluation

participants,                    too         few      people         would             participate                         in      the         follow-up             to

make       the     results                 valid.

           We believe                  USDA canceled                      the      follow-up                        study          prematurely.                      We

base       this         conclusion                  on     five       factors:                    four         of         which          are
specifically                   related              to     the      National                 WIC Evaluation                              and one which

is    related            to     the         experience               of     follow-up                      researchers                      in        general.

Specifically                   (1)      the         contractor              updated                 a large                 number               of    names

and      addresses              in      a short             period          of         time,             (2)         USDA did               not        contact
even       a sample             of      the         participant                  population                         to     determine                  their
interest           in     participating                       in     the         follow-up,                         (3)      less         than         1 year

had     elapsed               since           data        were        last            collected                  from         participants,                        which

compares           favorably                     with        past         success               in        follow-up                 research                done         by

us and         others              after          lapses            of        up to         20 years,                      and      (4)         as late            as

1987,       research                 analysts                working                for    the            Ford         Foundation                    reviewed              a

revised           proposal                 for       a follow-up                      study          of         the        same group                 of

children,               that         were         included                in        the     1983            proposal,                 and           concluded
that       the     study             was         feasible                and merited                       funding.                 The         other           factor,

related           to        general              follow-up                research,                   is        that         stronger                participant

interest           usually                 results            in         higher           participation                            rates.             We

believe           that          the        research                topic        --WIG's               effect               on the           cognitive

development                   of      their          children             --would               be of            strong             interest                to     the

participants,                       and thus               likely              to     produce               higher            participation.

           USDA's             estimate               of      the         likely           number                of     participants                        was based

on     updating               the       names,             addresses,                     and phone                   numbers             of        National             WIC

Evaluation                  participants.                          Using             methods               specified                by USDA--which

included               updating               addresses                  by     telephone                   or        mail         contacts                or

through           close             friends             or       relatives--in                        6 weeks                the      Research
Triangle               Institute                 updated             the        names,               addresses                   and phone                 numbers
of     about           82 percent                  of      those           eligible                  to     participate                        in    the         follow-

up study.                   However,               USDA estimated,                          with            no empirical                        basis,            that

only        the        45 percent                  of      the       sample               whose            addresses                 were           verified              by

 the    mail           or      telephone                  would          participate                       in        the     follow-up                  study.                In

our       opinion,                 since         USDA's             contractor                  had          updated               names            and

 addresses               of        82 percent                of      the            potential                participants,                          USDA,

 through          the          contractor,                   could             have        contacted                       these       people              to

accurately              determine                 their            willingness                    to     participate                in     the


CURRENT EFFORTS TO ASSESS WIC'S                                               IMPACT


           Because             of    the       continuing                    interest              of        the      National            Association

of    WIC Directors,                       Members            of       Congress,                  and others               in    determining

WIG's       impact             on children,                   USDA is               currently                  evaluating                the

feasibility               of        conducting                   a 5-l/2            to     6-l/2              year      follow-on               study          of

an entirely               new group                  of      WIC participants.                                 This      study           would            be

more       comprehensive                      than         the        one      proposed                 in     1983      and would                   address
WIG's        effect            on    anemia,               diet,           health           care,             and mental            and physical

development                in       children.                    It    would             also      make            comparisons                 to

determine,               for        example,               whether            WIC benefits                         some groups             more            than

others         and whether                    WIC participation                             is         more        effective             during

pregnancy,               infancy,              or         childhood.

           USDA staff                estimate                that          such          a study              would      cost        $16-22

million.              Funding              this           study        would             affect          other          WIC research                      unless

the      current           $3 million                     annual            legislative                      funding       ceiling                  for    WIC

research           is      raised.                USDA requested                          a temporary                   raise        in        WIC

research           funding              for         fiscal            year         1990          only.

           Because              USDA has             some unresolved                             concerns              about     both               the
 technical              feasibility--                     such        as     the         ability              to     assemble            a non-WIC

control          group --and                     the        proposed                 costs             of         conducting                      the         child

impact       study,             it         awarded                 a $652,000                     contract,                in        September                        1989,

to     conduct          field              tests             to     address               these                  concerns.                  USDA could

conclude          as early                  as         January             1991            that             it     is     impractical                           to     find      a

non-WIC          control                  group         and         thus         decide                not          to    undertake                         a child

impact       study.                  If     recruitment                        of     the          control                group                 is      practical,

further          evaluation                      of     other             design                 and        cost         issues                 will          take       until

August       1991         and             cost         an additional                             $635,000.                 At         that              time          USDA

could       decide          whether                    to         pursue            the          study.

FACTORS TO CONSIDER                                   IN FUTURE

MAJOR EVALUATIONS                            OF THE WIC PROGRAM

           Because          a study                    of         WIG's         impact              on children                       is             likely           to be      an

expensive              effort              with             significant                     policy                impacts,                 it          is      important

that       USDA not             repeat                 the         mistakes                 it      made during                       the              National            WIC

Evaluation.                 In            undertaking                      large            WIC research                        in         the              future,        USDA

should        take         steps             to        (1)         keep         the         study's                 implementation                              within

time       and     financial                     constraints,                         (2)         expedite                the         administrative
review       process                 to      communicate                        results                 to         the    Congress                          and other

interested              parties                  in         a more          timely                fashion,                and         (3)              ensure           that

production               and distribution                                 of        the          report             match            the             study's

significance                and             interest                 to        the        public.                   USDA can                    address               these
 factors         by:

          --   establishing                  realistic           time         goals         for         research,

               analysis,              and         reporting       on      study        results                as well          as

               stabilizing                  the      study's      design          earlier                in    the      research


          --   setting         forth              any    reservations             that            USDA may have                  with

               study       conclusions                   in    a separate             letter             rather         than

               rewriting              the         research       team's          summaries                and

               conclusions:                  and

          --   making         early          decisions           on     the     number             of     report         copies

               to     be printed              based           on the     needs         of         legitimate

               audiences           and        obtaining           advanced             approval                for

               expedited           or        commercial           printing             services                if     printing

               time      is    critical.

      This          concludes           my prepared               remarks.               We would                   be happy        to
respond        to     any questions                     you    may have.