United States General Accounting Office Testimony For Release on GAO's Review of USDA's National WIC Evaluation Delivery Report and Follow-up Issues Expected at 9:00 a.m. EDT Wednesday January 24, 1990 Statement of Keith 0. Fultz Director of Planning and Reporting Resources, Community, and Economic Development Division Before the House Select Committee on Hunger; the Senate Committee on Agriculture, Nutrition, and Forestry; and the Subcommittee on Nutrition, and Investigations, Senate Committee on Agriculture, Nutrition, and Forestry “-‘ \ dqf; 3; ’ , c..&)ij&-q GAO/T-RCED-90-21 GAO Form160<12/87) Messrs. Chairmen and members of the Committees: We are pleased to discuss our report on the Department of Agriculture's (USDA) handling of the National WICl Evaluation, as well as its research on the program's impact on children.* This report was reguested by the Chairman of the Select Committee on Hunger, Senators Tom Harkin and James Jeffords, and Representative Augustus Hawkins. In summary, USDA deleted the original chapter and executive summaries of the National WIC Evaluation and replaced them with its compendium of results. We believe that the original executive summary used appropriate methodology, was accurately presented, and pointed out the positive effects attributable to the WIC Program. In contrast, USDA's compendium contained errors and misleading statements about some of the data and deleted the study team's overall conclusions regarding the WIC Program's impact on participants. We also found that a combination of factors caused the National WIC Evaluation report to take 3-l/2 years longer and cost $2 million more than estimated to complete. For example, 'Special Supplemental Food Program for Women, Infants, and Children. *Our report entitled Food Assistance: The National WIC Evaluation: Reporting and Follow-up Issues (GAO/RCED-90-3, Dec. 14, 1989) is being publicly released today. about 2 years into the study, USDA replaced the study's principal investigator and redesigned the study. In 1983, USDA proposed another study--estimated to cost about $2 million-- to determine WIG's effects on children of the participants in the National WIC Evaluation. USDA withdrew the proposal prematurely in 1984. In 1987 USDA initiated efforts to determine the feasibility of conducting a more comprehensive child impact study on an entirely new sample population. In 1989 USDA staff estimated that the study proposed by its contractor will cost $16-22 million and take 5-l/2 to 6-l/2 years to complete. USDA currently has a contract to address its concerns about the technical feasibility and proposed costs of the new child impact study. We believe that it is important for USDA, when conducting major studies of WIG's impact, to learn from the mistakes made in the National WIC Evaluation. The balance of my testimony will provide you with a brief background on the National WIC Evaluation and address the flaws in USDA's compendium of results, the factors that delayed release of the National WIC Evaluation, the premature cancellation of the follow-up study on WIC children, the current efforts to assess WIG's impact on child development, and some factors to consider in future evaluations of the WIC Program. 2 BACKGROUND The WIC Program is a federally funded nutrition assistance program that provides supplemental foods, nutrition education, and referral to health care to low-income pregnant, breast-feeding, postpartum women: infants: and children up to 5 years old. Although USDA has undertaken many studies of specific aspects of the WIC Program, the National WIC Evaluation, which began in 1979 and was released in 1986, was the most complex and comprehensive evaluation of the program. This congressionally mandated study was USDA's second attempt to generate useful information on the effectiveness of the WIC Program. The first attempt-- commonly known as the Endozien study, after its principal investigator--was, according to USDA and others, severely criticized by researchers working in the field because of its research methodology. As a consequence, that study's results generally lacked credibility with the scientific community. To avoid the credibility problems of the Endozien study, USDA and its National WIC Evaluation study team had to carefully design and conduct the research to produce a credible study which would withstand the scrutiny of the scientific community. The end result of the National WIC Evaluation was a report which established WIG's effectiveness in several areas including: (1) improving the diet of pregnant women and children, (2) 3 lengthening gestation and reducing the likelihood of pre-term deliveries, and (3) increasing weight gain and the use of prenatal care in pregnant mothers. However, almost immediately after the study's release, the principal investigator for the study team and Members of Congress raised concerns regarding the manner in which the study was reviewed, revised, and released by USDA. USDA'S COMPENDIUM HAD SERIOUS FLAWS One of the areas that concerned the principal investigator and Members of Congress was the compendium of results which was written by USDA and substituted for the executive and chapter summaries written by the research team. We compared USDA's compendium of results and the research team's executive summary with the 5-volume National WIC Evaluation report. While we found only one minor error in the research team's summary, we found substantial flaws in the compendium which serve to understate--and mislead the reader about-- the generally positive effects of the WIC Program. Specifically, USDA's synthesis of the National WIC Evaluation's results included inappropriate methodological steps and reporting inaccuracies. There are four areas that could serve to understate the benefits of WIC participation and mislead the reader. 4 First, USDA did not preserve the original research design used by the team. The National WIC Evaluation was designed in a hierarchical fashion, with 6 major research issues divided into 15 subissues. Conclusions regarding the subissues were made by examining certain indicators (or measures) of those subissues. USDA's compendium eliminated this hierarchy and used categories created by USDA to report results. Some of these categories used National WIC Evaluation subissues, others combined two subissues or created new ones. Thus the compendium represented USDA's analysis of the National WIC Evaluation rather than a summary of the evaluation team's findings. This difference was not reported to the reader, consequently the reader is misled to assume that the compendium was a summary of the findings of the National WIC Evaluation. Second, USDA's compendium provided summary statistics that were not accurate. For example, USDA condensed results of over 100 measurements related to diet and reported that there were only 5 indications of diet assessed, thus generally understating WIG's impact on program participants. Third, USDA incorrectly reported conflicts in the significance of outcomes reported by the National WIC Evaluation. For example, USDA's compendium reports a conflict between the findings of the Historical Study of Pregnancy Outcomes and the 5 Longitudinal Study of Pregnant Women--two of the four component studies of the National WIC Evaluation--regarding WIG's impact on late fetal death. We found that there were no conflicts in the results of these studies, merely a difference in the statistical significance of the effects reported. The compendium's treatment of this issue leads the reader to believe that the Evaluation was unable to determine the WIC Program's impact on late fetal death. The National WIC Evaluation actually found a significant reduction of 2.3 deaths per 1,000 attributable to the WIC program. In essence by not reporting the conclusions reached by the research team, the compendium deprives the lay reader of access to the bottom-line findings of the National WIC Evaluation, and thus misleads the reader. Finally, USDA incorrectly reported the Evaluation's finding on health services by omitting certain indicators. For example, USDA did not report the significant impact that WIC benefits had on attaining a "regular source of medical care" for three age groups. Such omissions may lead a reader to assume that the researchers found a much weaker WIC effect than was actually the case. SEVERAL FACTORS DELAYED RELEASE OF THE NATIONAL WIC EVALUATION Although USDA estimated that the National WIC Evaluation would be completed in 2-l/2 years and cost about $3.9 million, the 6 study took about 6 years and cost $5.9 million. Release of the study was delayed for four principal reasons: (1) USDA officials replaced the study's principal investigator and the new investigator redesigned the study, (2) the research team had difficulty producing a product acceptable to USDA officials and the study's advisory panel within the time frames estimated, (3) USDA's review was protracted because it wrote a compendium of results to replace the study team's executive summary, and (4) unforseen printing problems were encountered. The National WIC Evaluation began in September 1979 with award of a contract to the Research Triangle Institute. To avoid the acceptance problems USDA encountered with the Endozien study, USDA established and used a panel to advise it on study design, scope, and report presentation. The panel, consisting of outside experts, included expertise in WIC administration, medical research, and research methodology. After reviewing the design, which was submitted for approval about 2 years after the study began, the advisory panel recommended that USDA redesign the study and place it under the direction of a medical researcher. As a result, USDA selected Dr. David Rush as principal investigator. Dr. Rush redesigned the study in 9 months and began his field work in the summer of 1982. Most of the data were collected by the end of 1983, although analysis of the data continued into late 1984. The first draft of the National WIC Evaluation, submitted in May 1984 to USDA and its advisory panel, was poorly written, according to 7 USDA, because the study team was attempting to analyze the data and draft reports on four component studies simultaneously. USDA changed the format of the report and gave the research team additional time to prepare a final product. The final draft of the National WIC Evaluation report, satisfying all of USDA staff's technical comments, was delivered to USDA in February 1985. The Administrator of the Food and Nutrition Service and the Assistant Secretary for Food and Consumer Services normally take about 8 weeks to review USDA reports for policy implications. However, the National WIC Evaluation took 9 months to review. The review was protracted because the chapter and executive summaries written by the research team were deleted in the Office of the Assistant Secretary for Food and Consumer Services and were replaced with the compendium of results,'which USDA staff wrote. USDA told us that it replaced the research team's summaries with the compendium because it believed that the summaries misstated the results of the National WIC Evaluation. Finally, in attempting to expedite the printing of the report, USDA misinterpreted the Government Printing and Binding Regulations. This resulted in the Department printing a different number of copies of each volume of the report and only 50 copies of the entire report. The quantities printed were insufficient, given the amount of public interest in the study. Release of the final 8 / report was further delayed because the initial printing was unreadable, and a second printing was required. USDA PLANNED THEN PREMATURELY CANCELED A FOLLOW-UP STUDY While the National WIC Evaluation was in progress, researchers found indications that WIC improved head size and perhaps brain growth and behavioral and cognitive development in children. These issues were outside the scope of the original study. However, USDA was convinced that these potential beneficial impacts on children merited further examination and therefore planned to conduct a follow-up study of the children born to mothers who participated in the National WIC Evaluation. USDA solicited such a study in 1983 but withdrew the request in 1984 citing its concern that, because of a confidentiality pledge given to National WIC Evaluation participants, too few people would participate in the follow-up to make the results valid. We believe USDA canceled the follow-up study prematurely. We base this conclusion on five factors: four of which are specifically related to the National WIC Evaluation and one which is related to the experience of follow-up researchers in general. Specifically (1) the contractor updated a large number of names and addresses in a short period of time, (2) USDA did not contact even a sample of the participant population to determine their interest in participating in the follow-up, (3) less than 1 year 9 had elapsed since data were last collected from participants, which compares favorably with past success in follow-up research done by us and others after lapses of up to 20 years, and (4) as late as 1987, research analysts working for the Ford Foundation reviewed a revised proposal for a follow-up study of the same group of children, that were included in the 1983 proposal, and concluded that the study was feasible and merited funding. The other factor, related to general follow-up research, is that stronger participant interest usually results in higher participation rates. We believe that the research topic --WIG's effect on the cognitive development of their children --would be of strong interest to the participants, and thus likely to produce higher participation. USDA's estimate of the likely number of participants was based on updating the names, addresses, and phone numbers of National WIC Evaluation participants. Using methods specified by USDA--which included updating addresses by telephone or mail contacts or through close friends or relatives--in 6 weeks the Research Triangle Institute updated the names, addresses and phone numbers of about 82 percent of those eligible to participate in the follow- up study. However, USDA estimated, with no empirical basis, that only the 45 percent of the sample whose addresses were verified by the mail or telephone would participate in the follow-up study. In our opinion, since USDA's contractor had updated names and addresses of 82 percent of the potential participants, USDA, through the contractor, could have contacted these people to 10 accurately determine their willingness to participate in the follow-up. CURRENT EFFORTS TO ASSESS WIC'S IMPACT ON CHILD DEVELOPMENT Because of the continuing interest of the National Association of WIC Directors, Members of Congress, and others in determining WIG's impact on children, USDA is currently evaluating the feasibility of conducting a 5-l/2 to 6-l/2 year follow-on study of an entirely new group of WIC participants. This study would be more comprehensive than the one proposed in 1983 and would address WIG's effect on anemia, diet, health care, and mental and physical development in children. It would also make comparisons to determine, for example, whether WIC benefits some groups more than others and whether WIC participation is more effective during pregnancy, infancy, or childhood. USDA staff estimate that such a study would cost $16-22 million. Funding this study would affect other WIC research unless the current $3 million annual legislative funding ceiling for WIC research is raised. USDA requested a temporary raise in WIC research funding for fiscal year 1990 only. Because USDA has some unresolved concerns about both the technical feasibility-- such as the ability to assemble a non-WIC 11 control group --and the proposed costs of conducting the child impact study, it awarded a $652,000 contract, in September 1989, to conduct field tests to address these concerns. USDA could conclude as early as January 1991 that it is impractical to find a non-WIC control group and thus decide not to undertake a child impact study. If recruitment of the control group is practical, further evaluation of other design and cost issues will take until August 1991 and cost an additional $635,000. At that time USDA could decide whether to pursue the study. FACTORS TO CONSIDER IN FUTURE MAJOR EVALUATIONS OF THE WIC PROGRAM Because a study of WIG's impact on children is likely to be an expensive effort with significant policy impacts, it is important that USDA not repeat the mistakes it made during the National WIC Evaluation. In undertaking large WIC research in the future, USDA should take steps to (1) keep the study's implementation within time and financial constraints, (2) expedite the administrative review process to communicate results to the Congress and other interested parties in a more timely fashion, and (3) ensure that production and distribution of the report match the study's significance and interest to the public. USDA can address these factors by: 12 -- establishing realistic time goals for research, analysis, and reporting on study results as well as stabilizing the study's design earlier in the research process: -- setting forth any reservations that USDA may have with study conclusions in a separate letter rather than rewriting the research team's summaries and conclusions: and -- making early decisions on the number of report copies to be printed based on the needs of legitimate audiences and obtaining advanced approval for expedited or commercial printing services if printing time is critical. This concludes my prepared remarks. We would be happy to respond to any questions you may have. 13
GAO's Review of USDA's National WIC Evaluation Report and Follow-up Issues
Published by the Government Accountability Office on 1990-01-24.
Below is a raw (and likely hideous) rendition of the original report. (PDF)