Systematic review of research methods: the case of business instruction

Abstract

Purpose

The purpose of this paper is to assess the body of business instruction literature by academic librarians against evolving models for evidence‐based research.

Design/methodology/approach

The paper used systematic review and inter‐rater reliability of the literature of business information research instruction to test two attributes of research quality: the evidence‐based levels of evidence and the EBLIP critical analysis checklist.

Findings

Intervention questions and case studies are the most popular research methods on the EBL levels of evidence scale. The majority of articles score below 75 on the EBLIP critical appraisal checklist. Prediction questions are represented by higher levels of evidence and study quality. Intervention questions paired with the cohort design and exploratory questions paired with survey design indicate strong areas of research quality. The case study method, while most popular, showes lower scores across all question types yet revealed some high‐quality benchmark examples.

Research limitations/implications

Error is possible when distinguishing between cohort and case study – some articles may fall into one or the other study design. Rater training was conducted only once, and best practices for inter‐rater reliability recommend multiple rounds to achieve higher rater agreement.

Practical implications

Recommendations are presented for ways to improve the evidence base of research articles and suggest areas for professional development opportunities for librarian researchers wishing to increase the quality of research publications.

Originality/value

The paper goes beyond the narrative review of the literature of business instruction to measure the research methods employed in those publications against two evidence‐based standards. The results will show where the literature stands as a maturing discipline and provide recommendations for increasing the levels of evidence for future research.

Citation

Manning Fiegen, A. (2010), "Systematic review of research methods: the case of business instruction", Reference Services Review, Vol. 38 No. 3, pp. 385-397. https://doi.org/10.1108/00907321011070883