To read this content please select one of the options below:

Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies

Tianji Jiang (Department of Information Studies, School of Education and Information Studies, University of California Los Angeles, Los Angeles, California, USA)
Jiqun Liu (School of Library and Information Studies, University of Oklahoma, Norman, Oklahoma, USA)

Aslib Journal of Information Management

ISSN: 2050-3806

Article publication date: 19 December 2022

Issue publication date: 2 January 2024




The purpose of this paper is to understand how users behave and evaluate how systems with users are essential for interactive information retrieval (IIR) research. User study methodology serves as a primary approach to answering IIR research questions. In addition to designing user study procedures, understanding the limitations of varying study designs and discussing solutions to the limitations is also critical for improving the methods and advancing the knowledge in IIR.


Given this unresolved gap, we apply the faceted framework developed by Liu and Shah (2019) in systematically reviewing 131 IIR user studies recently published (2016–2021) on multiple IR and information science venues.


Our study achieve three goals: (1) extracting and synthesizing the reported limitations on multiple aspects of user study (e.g. recruitment, tasks, study procedures, system interfaces, data analysis methods) under associated facets; (2) summarizing the reported solutions to the limitations; (3) clarifying the connections between types of limitations and types of solutions.

Practical implications

The bibliography of user studies can be used by students and junior researchers who are new to user-centered IR studies as references for study design. Our results can facilitate the reflection and improvement on IR research methodology and serve as a checklist for evaluating customized IIR user studies in varying problem spaces.


To our knowledge, this work is the first study that systematically reviews the study limitations and solutions reported by IIR researchers in articles and empirically examines their connections to different study components.



This work is partially supported by a Faculty Investment Program grant from the Research Council of the University of Oklahoma Norman Campus. The authors would like to thank Julie Torgerson and Shuyu Lu for their work on the initial-round paper annotation.


Jiang, T. and Liu, J. (2024), "Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies", Aslib Journal of Information Management, Vol. 76 No. 1, pp. 104-131.



Emerald Publishing Limited

Copyright © 2022, Emerald Publishing Limited

Related articles