-------- ---- COMPTROLLER GENERAL OF THE UNITED SATES WASHINGTON. D3.C. 20548 / B-133183 llnlllllllllllllllllllllllllll LM101732 March 4, 1977 The Honorable Ray Thornton Chairman, Subcommittee on Science, Research, and Technology 1 Committee on Science and Technology House o'f Representatives Dear Mr. Chairman: In accordance with the November 7, 1975 request of former Subcommittee Chairman James W. Symington and subsequent agree- ments with the Subcommittee office, we monitored the study jointly commissioned by the Subcommittee and the National / I ,: Science Hoard to-obtain information through questionnaires about the scientific community's views of the National Science Foundation peer review process; In November 1975, the Foundation mailed questionnaires to a random sample of 1,552 individuals, selected from its file of about 31,000 individuals, who had reviewed a research proposal for the Foundation in fiscal year 1974. Question- naires were also mailed to 3,256 individuals who were randomly selected from the Foundation's file of applicants who submit- ted about 20,000 proposals which were awarded, declined, or withdrawn in fiscal year 1975. The questionnaires asked the reviewers and applicants their experiences with the Founda- tion's proposal review process, their opinions of the review procedures, and their feelings regarding various possible modifications. Dr. Deborah R. Hensler L/ was employed by the Foundation as a private consultant to assist in the survey design by analyzing the responses to the questionnaires and reporting the results to the Subcommittee and the National Science Hoard. &/Dr. Hensler‘has a Ph.D in Political Science from the Massachusetts Institute of Technology and is an Associate Head of Rand Corporation's Social Science Department with primary responsibility for coordinating survey research. HRD-77-67 . B-133183 The scope of our work consisted of (1) maintaining the confidentiality of survey respondents' names by directly receiving the returned questionnaires and destroying the envelopes with the respondents' names, (2) determining that only those individuals selected to receive questionnaires were included in the study results by checking the returned questionnaires against the names included in the sample, (3) s verifying the accuracy of processing the original questionnaire responses into a computerized data file (master data file), and (4) verifying ,the statistics produced from the data as reported by Dr. Hensler. L/ Returned questionnaires were accepted through March 31, 1976, for inclusion in the study. The completed questionnaires for the reviewer and applicant surveys number 1,068 (69 per- cent response rate) and 2,684 (82 percent response rate), respectively. The responses were processed by a Foundation contractor (TeleSec) into a master data file containing the simple tabulated results. Prior to releasing the completed questionnaires to the Foundation, we reproduced responses to-questions at random from every second reviewer questionnaire and every third applicant questionnaire. The reviewer questionnaire had 34 questions, while the applicant questionnaire had 26 questions. The responses to questions which asked the respondent to pro- vide an explanation for his answer were not included in our sample for verification. We used this sample to check the accuracy of TeleSec's transferring the questionnaire responses to a computerized data file. We found no errors in the trans- fer of sampled responses for 33 of the 34 questions asked of reviewers and for 22 of the 26 questions asked of applicants. The following table shows the questions for which we found transfer errors, the projected number of questions in the universe for which errors were likely to exist, and the esti- mated error rates. L/Dr. Hensler reported the analysis of the questionnaire re- sponses in a December 1976 report entitled "Perceptions of the National Science Foundation Peer Review Process: A Report on a Survey of NSF Reviewers and Applicants." The report consists largely of 24 tables which present statistics created by analyzing the master data file. We independently verified the statistics, but did not evaluate Dr. Hensler's interpretation of the statistics. -2- ” 'B-133183 Projected no. of Estimated % of Questionnaire/ Universe questions with universe with Question No. size transfer errors transfer errors Reviewer 31 1,068 63 5.88 Applicant 2 2,684 69 2.56 8 2,684 69 2.56 10 2,684 96 3.57 21 2,684 84 .3.12 The Foundation created additional data from the responses to questions 31 and 32 on the reviewers' questionnaire, and questions 20 and 21 on the applicants' questionnaire. These questions concern the institutions which awarded the respon- dents their highest academic degrees, and the institutions with which respondents are currently affiliated. The Founda- tion categorized the reviewers' and applicants' degree-awarding institutions and their current affiliation by type, using an American Association of University Professors code-l/ Our ran- dom sample of the data base created from this process showed no errors for the coding of reviewers' institutions. However, for the coding of applicants' institutions, our sample showed estimated error rates of 1.02 percent for institutions award- ing the respondents their highest academic degrees and 2.04 percent for institutions with which respondents are currently affiliated. We believe that none of the errors in the master data file of questionnaire responses would greatly affect the tabu- lated resu,lts. In addition, it is highly probable that the errors are randomly distributed throughout the alternative choices of responses to a question which further reduces the chances to affect the tabulated results. We independently verified the statistics contained in Dr. Hensler's December 1976 report which were produced from L/Generally ranks participating institutions by categories, such as type and number of degrees awarded. I -3- . . c BL133183 the master- data file of questionnaire responses. We did not verify report tables 8, 13, 17, and 24 which contain respon- dents' explanatory data, and parts of tables 2 and 3 which contain data compiled by the National Science Foundation separate from the jointly-commissioned study. With these exceptions, we believe the statistics in the tables accurately present the master data file of questionnaire responses except for insignificant differences. Furthermore, the errors con- tained in the data base, as previously discussed, do not appear to greatly affect the statistics presented in the tables. We are available to discuss our findings and to provide any further assistance you might need in studying the Founda.- tion's peer review process. Sincerely yours, Comptroller General of the United States - 4 -'
Control of the National Science Foundation Peer Review Study
Published by the Government Accountability Office on 1977-03-04.
Below is a raw (and likely hideous) rendition of the original report. (PDF)