DOCUMENT RESUSE 03315 - [A1051741] Issues in Managing Applied Social Research. March 3, 1977. 8 pp. Staff study Conference of the Council for Applied Social Research, Washington, DC; by Keith I. Marvin, Associate Director, Program Analysis Div. Insue Area: Accounting and Financial Reporting (2800); Internal Auditing Systems: Sufficiency of Federal Auditors and Coverage (201). Contact: Program Analysis Div. Budget Function: General Government: Central Fiscal Operations (803). The General Accounting Office (GAO) has become increasingly involved in the review and audit of evaluative and social experimental efforts. The primary concern of GAO in this Lea has been to recognize the utility and effectiveness of the experimental methods employed while simultaneously assuring Congress that it will fulfill its role to invedtigate all matters relating to the receipt, disbursement, and application of public funds. While carrying out its responsibilities to Congress, GAO must, at the same time, minimize any problems created by audits of social experiments. GAO has identified 14 issues pertaining to the movement of the methods of social research into the policy process. GAO believes it should have access to all aspects of a contracted experimental program. The benefits of a more constructive public involvement in public policymaking is probably worth the cost of answering the resulting technical questions and issues. Answers to these questions need to be carefully considered by all those organizations and citizens who have an interest in building public accountability into the social oxperimentation process. (LDR) 'ISSUES IN MANAGING APPLIED SOCIAL RESEARCH BY KEITH E. MARVIN* ASSOCIATE DIRECTOR PROGRAM ANALYSIS DIVISION U.S. GENERAL ACCOUNTING OFFICE AT THE CONFERENCE OF THE COUNCIL FOR APPLIED SOCIAL RESEARCH WASHINGTON, D.C., MARCH 3, 1977 As a direct result of its historical mission to audit and review programs of the Federal Government, the General Accounting Office has increasingly become involved in the review and audit of evaluative and social experimental efforts. The perceived role of the GAO in this area of Governmental activity has been slowly building. Beginning with it's work in the New Jersey Work Incentive Experiment, and extended into the Performance Incentive Remedial Education Experiment, and most recently in the Experimental Housing Allowance Program, GAO has come to realize that it's participation in the experimental process involves it in some long-standing issues, issues previously thought to be solely the province of the experimental community. Early on, in the New Jersey experiment, it became apparent to the General Accounting Office, that the preliminary release of data from that experiment by the contracting agency raised questions about the right to access experimental data prior to the completion and the publication of experimental work done for the Federal Government. * The assistance of Dr. Stephen Baratz in this paper is acknowledged and appreciated, particularly his help in placing the issues and ideas in a timely science policy context. More recently, in the Experimental Housing Allowance Experiment, the General Accounting Office has become concerned and interested in the issue the audit and privacy of individually identified data gathered in support of the policy process in social experiments. The General Accounting Office has entered this area with a large measure of deliberation and care. It's major overriding concern has been to assure all parts of the interested public that it recognizes the utility and effectiveness of the experimental methods employed while it simultaneously assures the Congress of the maintenance of GAO's role to investigate all matters, such as accuracy and completer 's,relating to the receipt, disbursement, and application of public funds. Recognizing the significance of GAO's responsibility in this area, the Comptroller General, Elmer B. Staats, called together a meeting of expert and concerned citizens to explore with GAO people, how GAO can both meet it's responsibilities to the Congress while assuring that any problems created by audits of socidl experiments are minimized. Some of the Directors of the Council for Applied Social Research were among those who met with the Comptroller General on May 18, 1976. In anticipation of that meeting, the GAO conducted a considerable search of the literature on the experimental method and it's application to the public policy context. The results of that literature search revealed that there were at least fourteen separate sets of issues and concerns which could be identified as resulting from the movement of the methods of social research into the policy process (from bibliogrpahy listed at end). -2- For the purpose of my brief discussion with you today, I have distilled those fourteen sets of issues into three conceptual clusters. They are: 1) Issues relating to the ethical and legal constraints and trade-offs when social research moves into the policy process. Included ini this cluster of issues is the right to individual citizen privacy and the need for confidentiality of individually identified data, versus societies' right to know about that part of the universe covered by the ope ration of Government programs, particularly where the responsibility for public accountability is involved. This issue, recognized by researchers, contains elements of the precipitating issue which led to the May 18th meeting. 2) -Issues relating to the justification for, and the nature of design review procedures, as well as other technical problems of implementa- tion and management of social experiments. Incluaed in this cluster is the set of issues dealing with the need to consider alternative hypotheses and experimental procedures. 3) issues relating to the political, organizational and other practical problems of implementation and management of social experiments. Included here are issues relating to the need to consider alternative methods for the maintenance of quality control of social experiments. The May 18th meeting at the GAO identified an additional cluster of Issues on a Dar with the already identified fourteen issues. The May 18th meeting addressed but did not resolve all of the issues of how the society builds public accountability (as defined by GAO's mandate) into the social experimentation process. -3- GAO's opinion is that its mandate requires its access to all aspects of a contracted experimental program. The importance of this is indicated by the fact that our individual credentials which we carry con- t;in the key wow,' I referred to earlier "the Comptrollcr :'General shall investigate, at the seat of Government or elsewhere, all macters relating to the receipt, disbursement, and application of public funds. Perhaps Wilbur 'ampbell, Associate Director, in charge of the GAO review of the Housing Allowance Experiment, has put more thought than anyone else into how this applies to the audit of a large experiment. At the May 1976 nee+ing, he made the following statement, "I think the type of experiment we are talking about needs to be considered; and that is one where Fedcral funds are tUsed in experiments, paid to recipients for their use, and it impacts on public policy in that--depending upon the results of this experiment--it may result in a national program. It seems that this type of experiment is a little different from the experiment where an individual researcher goes out on his own and does research for some given purpose. Here you have Federal funds and public policy and it seems to us that the public has a right to know." This type of accountability, which implies much more thdn t:: 1t, research profession policing itself, must be considered of at least ce.. 1 importance to the other 14 sets of issues. This type of accountability extends concerns and issues relating to the development of procedures for methodological accountability, and the accountability for professionally competent treatments contained in an experiment. The literature reviewed is diverse with respect to how one resolves many of the issues raised. Ina sense, all of these issues apply to experiments on human subjects regardless of the size of the experiment. The methods of peer review and other institutional review of human experimentation have appeared.to be generally accepted and appear to have worked reasonably well when used in the way they were intended to be applied. No attempt is made in these remarks to propose specific refinements in those well established methods of managing human experimentation. This paper addresses the large social experiment and social research which is intended, by government contract, to be generalizable for public decisionmaking about the nature of future Government social programs and it is to that set of considerations tne paper now turns. In line with the topic of the present panel, Policies for Managing Applied Social Research, all important finding from an analysis of the identified issue clusters described previously Isthat a good deal of thought has gone into them; yet much or this thinking has been isolated in the various disciplines and sub-disciplines in the social and behavioral sciences without much cross-communication or cross-fertilization. For example, proper protection of detailed computerized con- fidential data involves the Information and the Computer Sciences. Cluarly, a number of different disciplines need to be involved in issues about design review procedures. The issues of accountability overlap the other issues and involve the Audit profession. Thus, one could conclude that it is vital that management of social experiments include a proper mix of appropriate disciplines. To properly -5- balance many of the issues identified above, it appears that the membership of these interdisciplinary teams, particularly those involved in the planning process of an experiment, should be publicly available. Also, the process by which they arrive at an acceptable, implementable, design for experimentation should be well documented as well as made available to the public concurrently. Auditing represents the public interest and thus the participation of the audit discipline symbolizes the nee. for broader public in- volvement in social experimentation intended to int!uence public policy. If policymaking is to be expanded to include the results of social experimentation then appropriate citizens need to be involved in a public process starting in the early planning stages of a social experiment; and throughout the different phases of the experiment itself. The Washington Post for today March 3, 1977, has an article describing a survey of 300,000 people, selected by the Census Bureau, to obtain their views relevant to alleviating the energy shortage, for consideration by the Administration in formulating energy policy. Opinions are to be obtained also from a large number of officials. I mention this only to indicate its feasibility, at modest cost relative to the cost of a major program. Could a more targeted, smaller survey approach be helpful, for example, in deciding the hypotheses which should be tested in major social experiments? Some may argue that it may be impossible to implement a major social experiment with this kind of visibility and openness. I do not agree. While there is a chance that it might delay or terminate some -6- proposals, I believe that this will be a temporary effect. The public education gained by such dissemination would benefit the planning and implementation process in futu:'e experiments, es well as provide a guarantee that the results would be used to the extent to which they are applicable in the decisionmaking process. With the type of openness described, I suspect that the Public would insist on no less. Questions relating to the above proposal come to mind immediately that need consideration by all those involved. Many questions have no answers that are immediately available. For example, would wide publicity given to an experiment in advance of its implementation invalidate the experimental design? In a more bureaucratic sense, would there be conflicts with existing Federal procurement regulations in selecting who is going to perform an experiment? The major conclusion of this paper is that the benefits of more constructive public involvement in this important area of public policy- making probably is well worth the cost of answering the kind of technical questions raised by this analysis. Answers to these questions need to be carefully considered and debated by all of those organizations and citizens who have interest and responsibilities in the public policy use of social experiments. -7- Bibliography 1. Adams, Stuart, Evaluative Reseagch in Corrections: A Practical Guide, Washington, D.C., U.S. Department cf Justice, 1975 2. Bernstein, Ilene Nagel and Howard E. Freeman, Academic and Entrepreneurial Research: The Consenuences of DiverSitv in Federal EvalJatiDn Studies, New York, Russell Sage, 1975 3. Committee on Federal Agency Evaluation Research, Assembly of Behavioral and Social Sciences, National Acaaemy of Sciences, Protecting Individual Privacy in Evaluation Research, Weshington, T.C.5, 195 4. Gotlieb, Allan, Charles Dalfen and Kenneth Katz, The Transborder Transfer .f Information bv Communications and Computer Systems: Issues rd Approacres to Guidina Prirciples, American Journal of Inter:national Law, Vol. 68, No. 2, April 1974 (pp. 227-257) 5. Hatry, Harry P., Richard E. Winnie and Donald M. Fisk, Practical Program Evaluation for State and local Government Officials, Washington, D.C., The Urban Institute, 1973 (pp. 56-6-2) 6. Horowitz, Irving Louis and James Everett Katz, Social Science and Public Policy in the United States, flew York, Praeger, 1975 7. Katz, Jay, Experimentation With Human Beings: The Authority of the Investigator, Subject, Professions, and State in the Human Experimentation Process, riew York, Russell Sage, 1972 8. Levin, Henry M., Some Methodoloaical Problems in Economic Polic Research: Determinin^ Hcw r:uch Should be SDent on Compensatory Education, Education and Urban Society, Vol. 7, N;o. 3, May 1975 (pp. 303-349) 9. Nagle, Stuart S., Ed. Policy Studies and The Social Sciences, Lexington, Mass., Lexington Books, 1975 10. Noble, John H., Jr., Peer Review: Quality Control of Applied Social Research, Science, Vol. 185, September 13, 1974 (pp. lb-92i) 11. Reicken, Henry W., and Robert F. Boruch, Ed. Social Experimentation: A Method for Planning and Evaluating Socia Intervention, New York, AEademic Press, 1974 12. Rivlin, Alice M. and P. Michael Timpane, eds., Ethical and Legal Issues of Social Experimentation, The Brookings Institution, 1975. 13. Struening, Elmer L., and Marcia Guttentag, Ed. Handbook of Evaluation Research, Vol. I and II, Beverly Hills, California, Sage Publications, 1975 14. Technology Assessment, Office of, Drug Bioequivalence, Washington, D.C., 1974
Issues in Managing Applied Social Research
Published by the Government Accountability Office on 1977-03-03.
Below is a raw (and likely hideous) rendition of the original report. (PDF)