Published February 19, 2024 | Version v1
Journal article Open

Facilitating and automating usability testing of educational technologies

Authors/Creators

  • 1. University of the Basque Country UPV/EHU

Description

Usability evaluation is a key element to ensure a positive user experience withany software and it is especially important in educational software tools wherethere are many different actors involved (lecturers, students, administrators,etc.). However, evaluating usability is not an easy task for nonexpertevaluators. To facilitate this evaluation task, this article presents a Methodol-ogy for Usability Testing (MUT) and a system (CALMUT) that assistsnonexpert evaluators in the application of the methodology by automatizingthe calculations and facilitating their interpretation. This can be very useful forlearning and instructional designers but also to people involved in the decisionof introducing or not a new educational software. To develop the proposal, aliterature review of different usability metrics, methods, and systems wascarried out first, followed by a selection and adaptation for novice usabilityevaluators. This article also presents a case study where lecturers tested theusability of an educational software following the proposal and shows thatusing MUT and CALMUT helps people without previous experience detect themain usability problems of educational systems before deciding whether to usethem or not.

Files

2024_CAEE.pdf

Files (2.3 MB)

Name Size Download all
md5:fdcb183566e8d30fb9865094c2fa4848
2.3 MB Preview Download

Additional details

Funding

Basque Government
Adian IT-1437-2
University of the Basque Country

References

  • American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Standards for educational and psychological testing, American Educational Research Association, Washington, DC, 2014.
  • I. D. Bleecker and R. Okoroji, Remote Usability Testing: Actionable insights in user behavior across geographies and time zones, Packt Publishing, Birmingham, UK, 2018.
  • J. Brooke, SUS: A retrospective, J. Usability Stud. 8 (2013), 29–40.
  • A. Bruun and J. Stage, New approaches to usability evaluation in software development: Barefoot and crowdsourcing, J. Syst. Softw. 105 (2015), 40–53. https://doi.org/10.1016/j.jss.2015.03.043
  • P. Buono, D. Caivano, M. F. Costabile, G. Desolda, and R. Lanzilotti, Towards the detection of UX smells: The support of visualizations, IEEE Access 8 (2020), 6901–6914. https://doi.org/10.1109/ACCESS.2019.2961768
  • L. Cayola and J. A. Macías, Systematic guidance on usability methods in user-centered software development, Inf. Softw. Technol. 97 (2018), 163–175. https://doi.org/10.1016/j.infsof.2018.01.010
  • S. Dimitrijevic and V. Devedzic, Usability evaluation in selecting educational technology, proceedings of 11th Conference of Information Technology and Development of Education (ITRO 2020), University of Novi Sad, Serbia, 2020.
  • S. M. Drucker, A. Glatzer, S. De Mar, and C. Wong, SmartSkip: consumer level browsing and skipping of digital video content, Proceedings of the SIGCHI conference on Human factors in computing systems Changing our world, changing ourselves - CHI '02, ACM Press, Minneapolis, Minnesota, USA, 2002, p. 219.
  • K. Finstad, The usability metric for user experience, Interact. Comput. 22 (2010), 323–327. https://doi.org/10.1016/j.intcom.2010.04.004
  • C. Gonzalez, Student usability in educational software and games: improving experiences, Information Science Reference, Hershey PA, 2013.
  • T. Granollers and J. Lorés, Incorporation of users in the Evaluation of Usability by Cognitive Walkthrough, HCI related papers of Interacción 2004 (R. Navarro-Prieto and J. L. Vidal, eds) Kluwer Academic Publishers, Dordrecht, 2006, pp. 243–255.
  • L. A. Hasan and K. T. Al-Sarayreh (2015) An Integrated Measurement Model for Evaluating Usability Attributes. In: Proceedings of the International Conference on Intelligent Information Processing, Security and Advanced Communication. ACM, New York, NY, USA, pp. 94:1-94:6
  • D. Hinderer and J. Nielsen, How to recruit participants for usability studies, Fremont, USA, 2003.
  • K. Hornbæk, Current practice in measuring usability: Challenges to usability studies and research, Int. J. Hum. Comput. Stud. 64 (2006), 79–102. https://doi.org/10.1016/j.ijhcs.2005.06.002
  • W. Hwang and G. Salvendy, Number of people required for usability evaluation: The 10±2 rule, Commun. ACM 53 (2010), 130–133. https://doi.org/10.1145/1735223.1735255
  • ISO, Ergonomics of human-system interaction—Part 11: Usability: Definitions and concepts, International Organization for Standardization, Geneva, Switzerland, 2018.
  • M. Y. Ivory and M. A. Hearst, The state of the art in automating usability evaluation of user interfaces, ACM Comput. Surv. 33 (2001), 470–516.
  • D. Karahoca, Meta-Cognitive tool development for history teaching: investigating how software usability affects student achievements, J. Univers. Comput. Sci. 19 (2013), 619–638. https://doi.org/10.3217/jucs-019-05-0619
  • D. Karahoca, Meta-Cognitive tool development for history teaching: investigating how software usability affects student achievements, J. Univers. Comput. Sci. 19 (2013), 619–638. https://doi.org/10.3217/jucs-019-05-0619
  • J. R. Lewis, Sample sizes for usability tests: Mostly math, not magic, Interactions 13 (2006), 29–33. https://doi.org/10.1145/1167948.1167973
  • J. R. Lewis, Measuring perceived usability: The CSUQ, SUS, and UMUX, Int. J. Human–Comput. Interact. 34 (2018), 1148–1156. https://doi.org/10.1080/10447318.2017.1418805
  • Loop11. https://www.loop11.com/, 2022.
  • B. Losada, M. Urretavizcaya, J-M López-Gil, and I. Fernández-Castro (2012) Combining InterMod agile methodology with usability engineering in a mobile application development. In: Actas de International Conference on Interacción Persona-Ordenador. ACM Press, Elche, p 39:1-39:8.
  • J. Lu, M. Schmidt, M. Lee, and R. Huang, Usability research in educational technology: A state-of-the-art systematic review, Educ. Technol. Res. Dev. 70 (2022), 1951–1992. https://doi.org/10.1007/s11423-022-10152-6
  • J. Lu, M. Schmidt, M. Lee, and R. Huang, Usability research in educational technology: A state-of-the-art systematic review, Educ. Technol. Res. Dev. 70 (2022), 1951–1992. https://doi.org/10.1007/s11423-022-10152-6
  • D. Marshall, J. C. Foster, and M. A. Jack, User performance and attitude towards schemes for alphanumeric data entry using restricted input devices, Behav. Inf. Technol. 20 (2001), 167–188. https://doi.org/10.1080/01449290110048007
  • MeasuringU, https://measuringu.com/calc/, 2022.
  • Nielsen Norman Group Nielsen Norman Group: UX Training, Consulting, & Research. In: Nielsen Norman Group. https://www.nngroup.com/, 2020.
  • D. Quiñones, C. Rusu, and V. Rusu, A methodology to develop usability/user experience heuristics, Comput. Stand. Interfaces 59 (2018), 109–129. https://doi.org/10.1016/j.csi.2018.03.002
  • J. Rubin and D. Chisnell, Handbook of usability testing: How to plan, design, and conduct effective tests, 2nd ed., Wiley Pub, Indianapolis, IN, 2008.
  • P. Runeson and M. Höst, Guidelines for conducting and reporting case study research in software engineering, Empir. Softw. Eng. 14 (2008), 131–164. https://doi.org/10.1007/s10664-008-9102-8
  • P. Runeson and M. Höst, Guidelines for conducting and reporting case study research in software engineering, Empir. Softw. Eng. 14 (2008), 131–164. https://doi.org/10.1007/s10664-008-9102-8
  • M. D. R. Sevilla-Gonzalez, L. Moreno Loaeza, L. S. Lazaro-Carrera, B. Bourguet Ramirez, A. Vázquez Rodríguez, M. L. Peralta-Pedrero, and P. Almeda-Valdes, Spanish version of the system usability scale for the assessment of electronic tools: Development and validation, JMIR Hum. Factors 7 (2020), e21161. https://doi.org/10.2196/21161
  • P. A. Smith, Towards a practical measure of hypertext usability, Interact. Comput. 8 (1996), 365–381.
  • D. S. Tan, G. G. Robertson, and M. Czerwinski (2001) Exploring 3D navigation: combining speed-coupled flying with orbiting. In: Proceedings of the SIGCHI conference on Human factors in computing systems—CHI '01. ACM Press, Seattle, Washington, United States, pp 418–425.
  • T. Tullis and B. Albert, Measuring the user experience: collecting, analyzing, and presenting usability metrics, Second edition., Elsevier/Morgan Kaufmann, Amsterdam; Boston, 2013.
  • T. Tullis and J. N. Stetson (2004) A comparison of questionnaires for assessing website usability. In: Proceedings of the 13th UPA Conference. Minneapolis. https://api.semanticscholar.org/CorpusID:9670323
  • usabiliTEST, Usability testing & Information Architecture. https://www.usabilitest.com/, 2020.
  • Usabilityscale, https://usabilityscale.com/, 2022.
  • M. Villamañe, B. Ferrero, A. Álvarez, M. Larrañaga, A. Arruarte, and J. A. Elorriaga (2014) Dealing with common problems in engineering degrees' Final Year Projects. In: Actas de IEEE Frontiers in Education Conference. IEEE Computer Society, Madrid, pp 2663–2670.
  • P. Vlachogianni and N. Tselios, Perceived usability evaluation of educational technology using the system usability scale (SUS): A systematic review, J. Res. Technol. Educ. 54 (2022), 392–409. https://doi.org/10.1080/15391523.2020.1867938
  • K. E. Wiegers and J. Beatty, Software requirements, Third edition, Microsoft Press, s division of Microsoft Corporation, Redmond, Washington, 2013.
  • P. Zaphiris and S. Kurniawan, Human computer interaction research in web design and evaluation, Idea Group Inc (IGI), Hershey, PA. USA, 2007.