A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of Web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform current models while avoiding as many of the biases of existing systems as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that, at least partially, resolves many of the technical and social issues associated with peer review, and can potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

A multi-disciplinary perspective on emergent and future innovations in peer review

Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments.

Communication is the Key Skill for Reference Librarians

A review of: Taylor, Robert S. "Question-Negotiation and Information Seeking in Libraries." College & Research Libraries 29.3 (1968): 178-94. Objective – To better understand the question negotiation process in libraries both in intermediated and in self-help situations. To achieve a richer understanding of the relationship between library users and library systems in order to establish a research agenda and inform librarian education. Design – The first part consisted of qualitative research involving interviews. The second part consisted of a diary study. Setting – Special engineering libraries in the United States and a university campus (Lehigh in Bethlehem, Pennsylvania). Subjects – The participants in the interviews were special librarians. Special librarians were selected because they have more specialized knowledge and respond to more substantive questions in greater depth than do public and academic librarians who emphasize instruction and who encounter staffing restrictions that prevent them from spending too much time on each inquiry. Detailed information on the selection of the individual participants is not provided. The participants in the diary study were twenty undergraduate students who were enrolled in an information science course. Methods – The interviews were open-ended and unstructured. The interviews lasted sixty to ninety minutes and were taped. No information is provided on transcription or analysis methods or paradigms. In the second part, the students were given a reading assignment on information seeking. They then had to select a search topic and document the steps they took, decisions they made, and resources they used to answer the question. The participants were asked to analyze their original question, the type of answer required, and decisions they made in the process. No details are provided on the analysis of the diaries. Main results – Taylor found five filters required for search definition: 1. Determination of subject; 2. Objective and motivation; 3. Personal characteristics of the inquirer; 4. Relationship of inquiry description to file organization; 5. Anticipated or acceptable answers (183) These five filters provide general information necessary for the for the search definition. These types are not mutually exclusive and may occur simultaneously. In the diary portion he found: 1. All participants consulted other people including librarians and fellow students; 2. None considered the library as a whole; 3. All inquiries required multiple sources; all answers were synthesized from multiple sources; 4. Participants were familiar with library research: they used the classification schedule to search, used subject headings, and used indexes or tables of contents. 5. Question or research problems changed as a result of information found Conclusion – Question negotiation is a dynamic process which requires feedback and iteration to come to a conclusion. The librarian’s job is to work with the inquirer to understand the information need and then to translate the negotiated need into appropriate search strategies. The author suggests that library school reference courses be updated to include instruction related to communication and negotiation in addition to the instruction on resources. He suggests more emphasis on questions instead of commands; that is, a cooperative process to determine what information is needed and how to best fulfill the need instead of assuming the inquirer “knows exactly what he wants, can describe its form (book, paper, etc.) and its label (author and title)” (191). To aid self-help situations, the author recommends better subject description of resources and inquiry-oriented instead of object-oriented systems. He suggests building better query negotiation into self-help systems. At minimum the system should request the user state his objective, if for no other reason than to force the user to reflect on or analyze his question. Help should be available at the time of need, and this can be offered through technology instead of through staffing.