Towards Standardized Pharmacology Examination Assessments: A Comparative Cross-University Study for Evaluation of Pharmacology Question Papers

Background: Standardized, valid, and reliable examination questions are essential to ensure fairness and to accurately assess competence in pharmacology. Variability in question construction, cognitive level tested, blueprinting, and item quality can contribute to inconsistent outcomes and threaten the validity of high-stakes undergraduate and postgraduate assessments.

Objective: To evaluate and compare the quality, cognitive level, content coverage, and alignment with learning outcomes of undergraduate and postgraduate pharmacology question papers across multiple universities in India.

Methods: This cross-sectional comparative study analyzed 180 summative pharmacology question papers (120 undergraduate, 60 postgraduate) from six medical universities in India, spanning the academic years 2020–2023. Papers were sampled using purposive sampling to include end-of-term and university final examinations. A validated checklist derived from standard assessment frameworks (including item writing guidelines, Bloom’s taxonomy, and assessment blueprinting principles) was used to assess each paper on 10 domains: blueprint alignment, cognitive level distribution, item clarity, presence of errors, inclusion of higher-order thinking items, coverage of core curriculum, marking scheme transparency, use of clinical vignettes, fairness (bias), and overall construct validity. Two independent trained raters evaluated each paper; discrepancies were resolved by consensus. Quantitative data were summarized with descriptive statistics; inter-rater reliability was assessed using Cohen’s kappa. Comparative analyses between universities and between undergraduate and postgraduate papers used chi-square tests and ANOVA where appropriate. Significance was set at p < 0.05.

Results: Overall, 38% of undergraduate papers and 62% of postgraduate papers met ≥ 7/10 quality criteria. Average cognitive level distribution favored recall (Bloom’s Level I–II) for undergraduate papers (mean recall proportion 68%), whereas postgraduate papers showed a higher proportion of application/analysis items (mean 47%; p < 0.001). Blueprint utilization was explicit in only 22% of papers. Item clarity errors (ambiguous stems, multiple correct options in MCQs, inconsistent options) were present in 29% of undergraduate and 15% of postgraduate papers (p = 0.02). Clinical vignette use was low in undergraduate assessments (14%) but common in postgraduate papers (53%). Inter-rater reliability for the checklist was substantial (κ = 0.78). Major gaps included poor blueprint adherence, insufficient higher-order item representation, inconsistent marking schemes, and variable curriculum coverage across universities.

Conclusion: Significant heterogeneity exists in pharmacology assessment quality across the studied universities. Undergraduate papers relied heavily on recall, with low blueprinting and limited clinical application items, while postgraduate papers performed better but still showed inconsistencies. The findings support the need for national guidelines, standardized item-writing training, peer review mechanisms, and shared blueprint templates to enhance assessment validity and fairness.