How Do Medical Students Approach Critical Appraisal? Results from a Mixed-Methods Study

Objectives: Our research team wanted to find out what principles and best practices medical students use when prompted by a clerkship assignment to complete a critical appraisal of an article of their choosing. Our hypothesis was that, outside of a structured classroom environment, many students would default to more basic literature evaluation strategies or even apply proxies for methodological rigor such as journal reputation or peer review status of a study.

Methods: All first-year clerkship students at the School of Medicine are required to complete a patient-focused evidence-based medicine (EBM) assignment during their Internal Medicine clerkship. A team of three librarians and one statistician undertook a mixed-methods approach to identify and quantify themes that emerged in the text of one year's worth of these assignments (n=343). A mixed method research approach was implemented to gain a greater understanding of the EBM principles and best practices that students reference in their assignments. Within this approach a qualitative content analysis was conducted, followed by a quantitative analysis of patterns within the sample. The research team used first- and second cycle coding and a collaboratively developed code list of nine major codes to ensure accuracy and standardization. Additionally, the research team's statistician implemented an inter-rater reliability plan and examined inter-class correlations to ensure grading consistency across team members and student assignments. Once all assignments had been coded, the team used statistical analysis to find correlations between codes as well as frequency of code application within the sample in order to identify five major critical appraisal themes which emerged in the students' assignments.

Results: After a rigorous coding process, several codes and related themes emerged. The research team identified nine main codes and five major themes. These themes are as follows:

Theme 1: Comparing the study population to the patient being treated and recommending a course of action
Theme 2: Identifying study type and position in hierarchy of evidence
Theme 3: Identifying proxies for study quality, including provenance and timeliness of chosen study
Theme 4: Summarizing study methodology and results
Theme 5: Attempting a critical appraisal of chosen study

Additionally, we identified correlations between themes as well as frequency of application in the sample.

Conclusions: A few notable results from our analysis of this sample are the frequency with which students were able to summarize the results of their chosen study and apply what they had learned to patient care (54.5% and 46.9% of all assignments, respectively). However, a notable number of students (35.2%) incorrectly used journal reputation, peer review status, h-index, impact factor, or similar metric, as a proxy for critical appraisal without engaging with the study methodology. This indicates that there is a need for further education and engagement with clerkship directors regarding the utility and application of EBM skills in the clinical curriculum.