Does students’ performance in the formative CLIPP examination predict their scores in the NBMEPSE?

Abstract

Purpose

Computer-Assisted Learning in Pediatrics Program (CLIPP) and National Board of Medical Examiners Pediatric Subject Examination (NBMEPSE) are used to assess students’ performance during pediatric clerkship. International Foundations of Medicine (IFOM) assessment is organized by NBME and taken before graduation. This study explores the ability of CLIPP assessment to predict students’ performance in their NBMEPSE and IFOM examinations.

Design/methodology/approach

This cross-sectional study assessed correlation of students’ CLIPP, NBMEPSE and IFOM scores. Students’ perceptions regarding NBMEPSE and CLIPP were collected in a self-administered survey.

Findings

Out of the 381 students enrolled, scores of CLIPP, NBME and IFOM examinations did not show any significant difference between genders. Correlation between CLIPP and NBMEPSE scores was positive in both junior (r = 0.72) and senior (r = 0.46) clerkships, with a statistically significant relationship between them in a univariate model. Similarly, there was a statistically significant relationship between CLIPP and IFOM scores. In an adjusted multiple linear regression model that included gender, CLIPP scores were significantly associated with NBME and IFOM scores. Male gender was a significant predictor in this model. Results of survey reflected students’ satisfaction with both NBMEPSE and CLIPP examinations.

Originality/value

Although students did not perceive a positive relationship between their performances in CLIPP and NBMEPSE examinations, this study demonstrates predictive value of formative CLIPP examination scores for their future performance in both summative NBMEPSE and IFOM. Therefore, students with poor performance in CLIPP are likely to benefit from feedback and remediation in preparation for summative assessments.

Citation

Almekhaini, L., Alsuwaidi, A.R., Alkaabi, K.K., Al Hamad, S. and Narchi, H. (2024), "Does students’ performance in the formative CLIPP examination predict their scores in the NBMEPSE?", Arab Gulf Journal of Scientific Research, Vol. 42 No. 2, pp. 359-369. https://doi.org/10.1108/AGJSR-11-2022-0255

Introduction

The College of Medicine and Health Sciences (CMHS) at the United Arab Emirates University (UAEU), a public federal university, offers a 6-year training program consisting of a 2-year premedical and a 4-year MD program that is divided into two phases of 2 years each: preclinical and clinical training, which center on different specialties. An 8-week pediatrics clerkship is offered in year five (junior clerkship), and a further 4-week clerkship in the final year (senior clerkship). Each clerkship program has a structured teaching program that includes bedside teaching in affiliated teaching hospitals. At the end of each clerkship program, students undergo an assessment that includes mini-clinical assessments (MiniCLEx) and the National Board of Medical Examiners Pediatrics Subject Examination (NBMEPSE).

The National Board of Medical Examiners (NBME) is an independent, nonprofit and web-based American organization established to develop countrywide examination regulations that allow medical accrediting authorities to evaluate candidates for licensing issuance. The NBME subject examination is used internationally to assess and train students in medical schools (Wright & Baston, 2017). It offers standardized and objective examinations in various disciplines, and these examinations are used throughout all phases of medical studies, particularly in the final assessments after clerkships (Johnson, Khalil, Peppler, Davey, & Kibble, 2014; National Board of Medical Examiners, 2016). Various studies have identified the NBME subject examination as the best intervention to evaluate low-performing students before graduation (Johnson et al., 2014; Guiot & Franqui-Rivera, 2018). It also serves as a global resource and a model for assessing approaches and appraisals in medicine, with a wide array of assessments that are well suited for medical schools (Guiot & Franqui-Rivera, 2018). These assessments allow medical schools to appraise their students’ achievement and compare their performance with a sizable, global and representative group of medical students at the same training stage (Bakoush, Al Dhanhani, Alshamsi, Grant, & Norcini, 2019). They assess both basic and clinical science knowledge (National Board of Medical Examiners, 2016) and comprise 100 multiple-choice questions (MCQs), each constructed in the form of clinical vignettes with one best answer that focuses on the application and integration of comprehension (National Board of Medical Examiners, 2019, 2020). Each of these examinations evaluates students’ comprehension of normal development (5%–10% of the questions), as well as a majority of cardiovascular, neurological, renal and other diseases, with each domain evaluated with 15% of the questions. These examinations also assess the important tasks of physicians, including health maintenance and promotion (5%–10% of questions), comprehension of disease mechanisms (25%–30% of questions), establishment of a diagnosis (40%–45% of questions) and the application of management principles (10%–15% of questions). The content area classification is as follows: foundation, diagnosis, management and prevention. The scores are compared across examinations, schools and examinee performance records. Individual performance profiles of correct percentage scores are reported. This web-based examination is administered in a secured setting in our college. In the final clerkship assessments administered, the NBMEPSE scores account for 30% of the total assessments (Narchi, 2013, 2014). In this study, NBME 1 refers to the examination offered at the end of the junior clerkship level (Year 1), while NBME 2 denotes the assessment administered at the end of the senior clerkship (Year 2). Students are provided with a score interpretation guide and a performance profile to facilitate their self-assessment (National Board of Medical Examiners, 2019, 2020).

In 2017, the Computer-Assisted Learning in Pediatrics Program (CLIPP) was added to both junior and senior pediatrics clerkships as a midterm formative assessment taken voluntarily. This is an online, validated assessment of 100-item MCQs created by the Council on Medical Student Education in Pediatrics (COMSEP), which is an independent organization of pediatric medical student education (Schifferdecker, Berman, Fall, & Fischer, 2012; Fall et al., 2005; Sox et al., 2018; COMSEP, 2020). The examination is delivered via a secure browser; it complements the performance assessment of students during their clerkship, enhances students’ self-directed learning and helps develop appropriate remediation when needed (Aquifer pediatrics, 2020). It also helps students prepare for the summative NBMEPSE at the end of their clerkship training. The students receive their CLIPP reports with their respective scores well before their NBMEPSE examination. In addition, the CLIPP reports also help ascertain the effectiveness of the curriculum as well as teaching techniques employed by instructors. In this study, CLIPP 1 refers to the examination administered during junior clerkship (Year 1) and CLIPP 2 to the one administered during senior clerkship (Year 2).

At the end of their last year of study, just before graduation, students sit for the International Foundation of Medicine (IFOM) examination, which covers all aspects of clinical medicine as opposed to the NBME Subject Examination and the CLIPP. The IFOM examination comprises 160 MCQs from basic clinical science in the following clinical areas: pediatrics, medicine, family medicine, surgery, psychiatry, and obstetrics and gynecology (IFOM, 2019a, b). In addition, it allows the comparison of CMHS-UAEU students’ performance prior to graduation with that of final-year medical students around the world. Therefore, the CMHS-UAEU uses the IFOM Clinical Science Examination as an essential part of assessing students prior to graduation. In this study, IFOM 2 refers to the examination administered in the final year prior to graduation.

The objectives of this study were as follows: (1) to evaluate, in each clerkship, the predictive value of the CLIPP formative assessment on students’ performance in the NBMEPSE summative assessment as well as on their performance in the final IFOM assessment and (2) to survey the students’ satisfaction and perceptions regarding the benefits of both of these evaluation tools. The null hypothesis was that the formative CLIPP scores do not predict the NBMEPSE or the IFOM scores. If that hypothesis was rejected for both independent summative assessments (NBMEPSE and IFOM) in both clerkships, the value of the formative CLIPP assessment to predict the two latter scores would be firmly validated. This would therefore suggest that identifying the students performing poorly in the CLIPP assessment would pave the way for the implementation of remediation measures that can improve students’ future performance in both the NBMEPSE and the IFOM.

Methods

This study, undertaken at the CMHS-UAEU, is a retrospective observational review of the marks obtained by all junior and senior students as well as students who recently graduated after completing both their voluntary formative CLIPP and the mandatory summative NBMEPSE at the end of each of the two pediatric clerkships between October 2017 and June 2020.

In addition, at the end of each NBMEPSE, students were asked to participate in a 5-min online survey via a self-administered questionnaire regarding their perceptions about the value of the CLIPP examination and any correlation they think it has with the NBMEPSE (the complete survey is available as Supplemental Digital Appendix 1). The survey was anonymous and did not have any impact on the students’ progression throughout their medical studies. By completing the online survey, the students confirmed their voluntary consent to participate in this study. Participants’ confidentiality was maintained, and no identifying data were included in data analysis. The participants were informed that they reserved the right to withdraw at any stage of the study without facing any penalties.

Statistical analysis

After testing for normality (Wilks–Shapiro test), the examination scores were expressed as mean value and standard deviation (SD). The group mean scores were compared using the unpaired t-test, and the correlation between the students’ CLIPP and NBMEPSE scores was tested using the Pearson product–moment correlation. The correlation between the CLIPP and NBMEPSE scores in each clerkship was tested using a univariate linear multivariate regression model, and the IFOM scores were tested using a linear multivariate regression model including gender. The results of the survey were expressed as percentages of students selecting a particular option on a 5-point Likert scale. All the statistical tests were performed using the STATA version 15 software package (StataCorp, Texas, USA), and a two-sided p value < 0.05 defined statistical significance.

Results

Participants

A total of 381 students (299 females, 78.5%) were enrolled during the study period. Out of these, 286 students had completed both junior and senior clerkships. In the junior clerkship, the CLIPP-1 assessment was taken by 200 students and the NBME-1 by 353 students, with 200 taking both examinations. The CLIPP-2 assessment and the NBME-2 assessment in the senior clerkship were taken by 223 and 286 students, respectively, with 223 taking both examinations. The IFOM-2 examination was taken by 167 senior students (43.8%). However, in total, only 128 junior and senior students completed the online survey.

Assessment scores

There were no significant differences in the CLIPP, NBME and IFOM scores based on gender (Table 1). Similarly, there were no significant differences in the NBME and IFOM scores between the students who voluntarily took the CLIPP assessment and those who did not (Table 2).

Correlation between the CLIPP and NBME scores

While there was a good correlation (r = 0.72) between both scores and in both genders in the junior clerkship, it was more modest (r = 0.46) in the senior clerkship where it was weak in females (r = 0.44) but remained strong (r = 0.73) in males (Table 3).

Modeling of the relationship between the CLIPP and NBME scores

In both clerkships, there was a statistically significant linear relationship between both scores (Table 4). In Junior Clerkship 1, for each increase by 1 mark in the CLIPP score, the resulting increase in the NBME mark was 0.82 (Table 3) regardless of gender. In Senior Clerkship 2, for each increase by 1 mark in the CLIPP score, the resulting increase in the NBME mark was 0.54, with an additional effect of gender; that is to say, the CLIPP scores of male students were expected to increase by 8.7 marks when compared with those of female students (Table 4).

Modeling of the relationship between the CLIPP and IFOM scores

In the unadjusted simple linear regression model, CLIPP 2 scores were significantly associated with the final IFOM 2 scores. However, in the adjusted linear regression model, gender was significantly associated with the IFOM scores. In the senior clerkship, for each extra mark in CLIPP 2 scores, the IFOM 2 score increased by 3.82 marks, with an additional 43.15 marks in male students’ scores when compared with female students’ scores (Table 4).

CLIPP survey

Most students were happy about the suitability of the setting and venue. They were also positive about the contents of the scenarios and questions, the authenticity of the questions, their clarity, the optimal use of software and the examination duration (Figure 1 and Table 5). However, many students did not feel that CLIPP prepared them well for the NBME or helped predict their performance in the NBME. They also felt that the feedback they received after the CLIPP assessment did not help them prepare for the NBME.

Discussion

This study confirms that both the summative NBME and IFOM scores increase in proportion to the formative CLIPP score, corroborating that both can be predicted by formative CLIP scores. This significant relationship between formative CLIPP scores and summative NBMEPSE scores, which is not surprising as both examinations assess knowledge in pediatrics, was also found between the CLIPP assessments – exclusively in pediatrics – and the final IFOM summative examinations, which evaluate knowledge in all branches of medicine. This strongly reinforces the strong predictive value of the formative CLIPP assessment regarding all summative examinations. However, this is not unexpected because these examinations assess students’ knowledge, also confirming the findings of previous studies demonstrating the positive correlation between the CLIPP examination and the summative NBME scores (Morrison et al., 2010). In addition, the use of its patient case simulations helps improving the teaching quality (Leong, Baldwin, & Adelman, 2003). Identifying poorly performing students in the formative CLIPP assessment and giving them appropriate feedback and remediation should, therefore, intuitively help the students improve their performance in the summative NBME and IFOM assessments.

Although this study employs CLIPP as a formative pediatric assessment, it is surprising that it also predicts the final IFOM examination that encompasses all branches of medicine. It is not expected that improving students’ knowledge in pediatrics, demonstrated in the CLIPP results, would enhance their knowledge in all other branches of medicine as tested in the IFOM. Therefore, we can speculate that the students’ scores in both CLIPP and IFOM examinations reflect the students’ own inherent learning abilities across a wide range of domains instead of attributing the enhancement of their IFOM scores directly and solely on their pediatric CLIPP experience.

Surprisingly, the students did not perceive any value of the CLIPP feedback in helping them prepare for the NBME. This could be attributed to the fact that they currently receive only their CLIPP scores without any detailed feedback on their performance, thus making it impossible to identify their weaknesses and take the necessary measures to improve their future performance. The participants believe that the CLIPP examination is easier than the NBME examination is as it focuses on a limited range of clear objective pediatrics conditions as opposed to the NBME examination, which covers a broad range of pediatric ailments and requires deeper and more advanced knowledge. This is perhaps the reason for which the students felt that the CLIPP examination had no significant role in improving their readiness for the NBME examination. However, they admitted that taking the CLIPP examination earlier in their clerkship helped them objectively benchmark their knowledge and identify their strengths and weaknesses. Consequently, this encouraged them to improve their knowledge.

In contrast to the plethora of reports in the literature evaluating the NBME and the United States Medical Licensing Examination (USMLE) (Boscardin, Earnest, & Hauer, 2020; Torre et al., 2020; Rosenthal et al., 2019), the paucity of publications regarding the relationship between the CLIPP assessment and other examinations in pediatric clerkships does not allow researchers to make any comparisons (Schifferdecker et al., 2012; Sox et al., 2018). This is probably due to the fact that CLIPP is an exclusive pediatric assessment as opposed to the NBMEPSE examinations, which cover all medical disciplines.

This study has some limitations. In addition to the small sample size of students, not all the students took the two formative CLIPP assessments, the two summative NBMEs or even the final IFOM assessment. In addition, the effects of the remediation provided to some students after their CLIPP assessment reports and just before the summative assessments were not analyzed because no systematic documentation existed. Finally, because the CLIPP examination was an optional and voluntary formative assessment, its outcomes could not be assessed in all students. However, the findings of this study showed no significant differences in the NBME and IFOM scores between students who voluntarily took the CLIPP assessment and those who did not. This implies that the proportionality between the CLIPP assessment scores and both summative assessments scores could be attributed to the students’ own intellectual capabilities as opposed to the feedback received from the summative assessments.

Future studies should consider using a larger sample size and conducting a formal analysis of the role of remediation measures implemented after the CLIPP on the summative assessment scores. Including the students’ NBME performance in other medical disciplines (which do not have CLIPP assessments) in the analysis of the final IFOM scores would also help better delineate the specific contribution of the CLIPP and NBMEPSE pediatric assessments to the IFOM.

Conclusion

This study reveals that the CLIPP examination is a good predictor of students’ performance in both the NBMEPSE assessments during pediatrics clerkships and the IFOM assessment prior to graduation. Therefore, giving students detailed feedback on their performance in the CLIPP examination is essential and should be an integral part of their educational experience. Appraising students assessment tools provides robust evidence-based information for improving their performance. Future studies should include larger sample size to evaluate the role of pre-assessment preparation, post-assessment feedback and appraisal of remedial measures to improve future performance. Introducing and examining a new assessment tool matched with appropriate elements of instructional design is imperative. These may include gamification, interface design, density of the content presented, in addition to the learning and cognitive styles of the learners.

Figures

Figure 1

CLIPP survey results

Table 1

Examination scores of students in the two clerkships

	Males	Females	p value*	All students
CLIPP 1	39.7 ± 11.9	39.1 ± 11.0	0.7	39.3 ± 11.2
NBME 1	51.0 ± 13.0	52.6 ± 1.9	0.2	52.3 ± 12.2
CLIPP 2	38.2 ± 13.5	42.6 ± 13.4	0.06	41.8 ± 13.5
NBME 2	56.1 ± 14.4	54.4 ± 14.5	0.4	54.8 ± 14.5
IFOM 2	529.6 ± 95.8	522.8 ± 82.6	0.6	524.3 ± 85.4

Note(s): n = 200 in junior clerkship and 223 in senior clerkship, expressed as mean ± standard deviation (SD) Assessments 1 and 2 represent junior and senior pediatric clerkships, respectively

*unpaired t-test

Source(s): Authors’ work

Table 2

Comparison of NBME and IFOM examination scores between students who voluntarily took the CLIPP assessment and those who did not

	CLIPP taken	CLIPP not taken	p value*
Total students	200	143
NBME 1	53.0 ± 12.7	51.2 ± 11.3	0.17
Total students	223	63
NBME 2	54.6 ± 14.9	55.5 ± 13.1	0.63
Total students’	151	16
IFOM 2	526.6 ± 86.4	502.3 ± 74.2	0.27

Note(s): The results are expressed as mean ± standard deviation (SD), unless stated otherwise Assessments 1 and 2 represent junior and senior pediatric clerkships, respectively

*unpaired t-test

Source(s): Authors’ work

Table 3

Pearson’s r correlation between the NBME and CLIPP scores in each of the junior and senior clerkships

	Males	Females	All students
Junior clerkship (Year 1)	0.72	0.72	0.72
Senior clerkship (Year 2)	0.73	0.44	0.46

Source(s): Authors’ work

Table 4

Multivariate linear regression modeling of NBME and IFOM scores from the obtained CLIPP scores, corrected for gender, in each of the junior and senior clerkships

	Unadjusted models*					Adjusted model§
	Students n	Coefficient for CLIPP (95% ci)	Constant	Adjusted R2	p value	Coefficient for CLIPP (95% ci)	Constant	Adjusted R2	p value	Coefficient for gender (p value)
NBME 1 (junior)	200	0.82 (0.71. 0.93)	20.79	0.52	<0.001	0.82 (0.71, 0.93)	20.4	0.52	<0.001	1.72 (0.22)
NBME 2 (senior)	223	0.51 (0.38, 0.63)	33.23	0.21	<0.001	0.54 (0.41, 0.66)	30.48	0.26	<0.001	8.78 (<0.001)
IFOM 2 (senior)	151	3.71 (2.9, 4.5)	380.13	0.36	<0.001	3.82 (3.0, 4.6)	369.31	0.39	<0.001	43.15 (0.006)

Note(s): * Simple linear univariate regression; ^§ multivariate linear regression model adjusting for gender

Predicted NBME 1 score = (0.82 × CLIPP 1 score) + 20.4 (p < 0.001)

Predicted NBME 2 score = (0.54 x CLIPP 2 score) + 30.48 + 8.78 if male student (p < 0.001)

Predicted IFOM score = (3.82 x CLIPP 2 score) + 369.31 + 43.15 if male student (p < 0.001)

Source(s): Authors’ work

Table 5

CLIPP examination survey results expressed as Likert score mean ± SD

	Mean ± SD
The CLIPP examination adequately assesses the paediatric clerkship’s learning objectives	3.5 ± 1.1
It adequately matches your academic-year level	3.5 ± 1.1
Its contents, including the scenarios and answer options, are clear, well written and up-to-date	3.7 ± 1.2
The cases are authentic and realistic	3.9 ± 1.0
It provides adequate preparation for the pediatric NBME	2.9 ± 1.2
It is a useful experience within the pediatric clerkships	3.5 ± 1.2
The feedback helped you predict your future NBME performance	2.8 ± 1.2
The feedback helped guide your learning objectives plan	3.2 ± 1.3
The feedback is a useful tool in preparing for pediatrics NBME	2.9 ± 1.3
It is important to include the examination results and the feedback in the pediatric curriculum	3.1 ± 1.2
Its timing was appropriate to support your preparation for the NBME assessment	3.0 ± 1.3
The allocated time was sufficient to answer all questions	3.7 ± 1.2
The examination software was used optimally	3.9 ± 1.1
The scenarios, questions and answer options were visually clear and suitable	4.0 ± 0.9
The CLIPP examination setting/venue was appropriate	4.1 ± 1.0

Source(s): Authors’ work

Supplemental Digital Appendix 1