Search results

During the literature review phase, the task of finding similar research papers can be a difficult proposition for researchers due to the procedural complexity of the task. Current systems and approaches help in finding similar papers for a given paper, even though researchers tend to additionally search using a set of papers. This paper aims to focus on conceptualizing and developing recommendation techniques for key literature review and manuscript preparatory tasks that are interconnected. In this paper, the user evaluation results of the task where seed basket-based discovery of papers is performed are presented.

A user evaluation study was conducted on a corpus of papers extracted from the ACM Digital Library. Participants in the study included 121 researchers who had experience in authoring research papers. Participants, split into students and staff groups, had to select one of the provided 43 topics and run the tasks offered by the developed assistive system. A questionnaire was provided at the end of each task for evaluating the task performance.

The results show that the student group evaluated the task more favourably than the staff group, even though the difference was statistically significant for only 5 of the 16 measures. The measures topical relevance, interdisciplinarity, familiarity and usefulness were found to be significant predictors for user satisfaction in this task. A majority of the participants, who explicitly stated the need for assistance in finding similar papers, were satisfied with the recommended papers in the study.

The current research helps in bridging the gap between novices and experts in terms of literature review skills. The hybrid recommendation technique evaluated in this study highlights the effectiveness of combining the results of different approaches in finding similar papers.

Systems to support literature review (LR) and manuscript preparation tend to focus on only one or two of the tasks involved. The purpose of this paper is to describe an intervention framework that redesigns a particular set of tasks, allowing for interconnectivity between the tasks and providing appropriate user interface display features for each task in a prototype system.

A user evaluation study was conducted on the prototype system. The system supports the three tasks: building a reading list (RL) of research papers, finding similar papers based on a set of papers and shortlisting papers from the final RL for inclusion in manuscript based on article type. A total of 119 researchers who had experience in authoring research papers, participated in the evaluation study. They had to select one of the provided 43 topics and execute the tasks offered by the system. Three questionnaires were provided for evaluating the tasks and system. Both quantitative and qualitative analyses were performed on the collected evaluation data.

Task redesign aspects had a positive impact in user evaluation for the second task of finding similar papers while improvement was found to be required for the first and third tasks. The tasks interconnectivity features seed basket and RL were helpful for the participants in conveniently searching for papers within the system. Two of the four proposed informational display features, namely, information cue labels and shared co-relations were the most preferred features of the system. Student user group found the task recommendations and the overall system to be more useful and effective than the staff group.

This study validates the importance of interconnected task design and novel informational display features in accentuating task-based recommendations for LR and manuscript preparatory tasks. The potential for improvement in recommendations was shown through the task redesign exercise where new requirements for the tasks were identified. The resultant prototype system helps in bridging the gap between novices and experts in terms of LR skills.

Although many interventional approaches have been proposed to address the apparent gap between novices and experts for literature review (LR) search tasks, there have been very few approaches proposed for manuscript preparation (MP) related tasks. The purpose of this paper is to describe a task and an incumbent technique for shortlisting important and unique papers from the reading list (RL) of researchers, meant for citation in a manuscript.

A user evaluation study was conducted on the prototype system which was built for supporting the shortlisting papers (SP) task along with two other LR search tasks. A total of 119 researchers who had experience in authoring research papers participated in this study. An online questionnaire was provided to the participants for evaluating the task. Both quantitative and qualitative analyses were performed on the collected evaluation data.

Graduate research students prefer this task more than research and academic staff. The evaluation measures relevance, usefulness and certainty were identified as predictors for the output quality measure “good list”. The shortlisting feature and information cues were the preferred aspects while limited data set and rote steps in the study were ascertained as critical aspects from the qualitative feedback of the participants.

Findings point out that researchers are clearly interested in this novel task of SP from the final RL prepared during LR. This has implications for digital library, academic databases and reference management software where this task can be included to benefit researchers at the manuscript preparatory stage of the research lifecycle.