3554722
doi
10.5281/zenodo.3554722
oai:zenodo.org:3554722
Pope, Denise S
SimBiotic Software Inc
Wendel, Daniel
Massachusetts Institute of Technology
Meir, Eli
SimBiotic Software Inc
WordBytes: Exploring an Intermediate Constraint Format for Rapid Classification of Student Answers on Constructed Response Assessments
Kim, Kerry J.
SimBiotic Software Inc
url:https://jedm.educationaldatamining.org/index.php/JEDM/article/view/209
info:eu-repo/semantics/openAccess
Creative Commons Attribution Non Commercial No Derivatives 4.0 International
https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode
student answers
assessment
constructed response
reliability
Computerized classification of student answers offers the possibility of instant feedback and improved learning. Open response (OR) questions provide greater insight into student thinking and understanding than more constrained multiple choice (MC) questions, but development of automated classifiers is more difficult, often requiring training a machine learning system with many human-classified answers. Here we explore a novel intermediate constraint question format called WordBytes (WB) where students assemble one-sentence answers to two different college evolutionary biology questions by choosing, then ordering, fixed tiles containing words and phrases. We found WB allowed students to construct hundreds to thousands of different answers ([?]20 tiles), with multiple ways to express correct and incorrect answers with different misconceptions. We found humans could specify rules for an automated WB grader that could accurately classify answers as correct/incorrect with Cohen's kappa [?] 0.88, near the measured intra-rater reliability of two human graders and the performance of machine classification of OR answers (Nehm et al., 2012). Finer-grained classification to identify the specific misconception had lower accuracy (Cohen's kappa < 0.75), which could be improved either by using a machine learner or revising the rules, but both would require considerably more development effort. Our results indicate that WB may allow rapid development of automated correct/incorrect answer classification without collecting and hand-grading hundreds of student answers.
The file is in PDF format. If your computer does not recognize it, simply download the file and then open it with your browser.
Zenodo
2017-12-23
info:eu-repo/semantics/article
3554721
1.0.0
1579527523.569166
1215115
md5:5da37095b73b085b5ccfbf56b313faac
https://zenodo.org/records/3554722/files/96543722
public
https://jedm.educationaldatamining.org/index.php/JEDM/article/view/209
Is cited by
url
10.5281/zenodo.3554721
isVersionOf
doi
Journal of Educational Data Mining
9
2
45-71
2017-12-23