Published September 5, 2024 | Version v1
Conference paper Open

Revolutionizing Qualitative Human-Robot Interaction Research by Using GPT Models for Inductive Category Development

Creators

  • 1. ROR icon Bielefeld University

Description

Coding qualitative data is essential but time-consuming. This late-breaking report presents a new method for developing inductive categories utilizing GPT models. We examined two different GPT models (gpt-3.5-turbo-0125 and gpt-4o-2024-05-03) and three temperature settings (0, 0.5, 1), each with ten repetitions. The generated categories were fairly consistent across settings, although higher temperatures included less relevant aspects. The agreement for GPT-generated category assignments exceeded that of human coders, with the best performance observed at temperature setting 0. Thus, we recommend using a GPT model with the temperature setting 0 to create and assign inductive categories for qualitative data.

Files

Arlinghaus_et_al_2024_ROMAN_LBR_GPT.pdf

Files (814.9 kB)

Name Size Download all
md5:2137a2c27415d8aa1e528bb34fdb2855
814.9 kB Preview Download

Additional details

Related works

Is part of
Preprint: 10.31219/osf.io/gpnye (DOI)

References

  • L. Veling, and C. McGinn, Qualitative research in HRI: A review and taxonomy, International Journal of Social Robotics, vol. 13, pp. 1689- 1709, February 2021. https://doi.org/10.1007/s12369-020-00723-z
  • P. Mayring, Qualitative content analysis, Forum: Qualitative Social Research, vol. 1, no. 2, article 20, June 2000. https://doi.org/10.17169/fqs-1.2.1089
  • N. Vollstedt, and S. Rezat, An introduction to Grounded Theory with a special focus on axial coding and the coding paradigm, in G. Kaiser and N. Presmeg (Eds.), Compendium for Early Career Researchers in Mathematics Education, ICME-13 Monographs, April 2019. https://doi.org/10.1007/978-3-030-15636-7 4
  • H.-F. Hsieh, and S. E. Shannon, Three approaches to qualitative content analysis, Qualitative Health Research, vol. 15 no. 9, pp. 1277- 1288, November 2005. https://doi.org/10.1177/1049732305276687
  • I. G. Raskind, R. C. Shelton, D. L. Comenau, H. L. Cooper, D. M. Griffith, and M. C. Kegler, A review of qualitative data analysis practices in health education and health behavior research, Health Education and Behavior, vol. 46, no. 1, pp. 32-39, February 2019. https://doi.org/10.1177/1090198118795019
  • S. P. Church, M. Dunn, and L. S. Prokopy, Benefits to qualitative data quality with multiple coders: Two case studies in multi-coder data analysis, Journal of Rural Social Sciences, vol. 34, no. 1, article 2, August 2019. https://egrove.olemiss.edu/jrss/vol34/iss1/2/
  • P. Baumgartner, A. Smith, M. Olmsted, and D. Ohse, A framework for using machine learning to support qualitative data coding, Open Science Framework, November 2021. https://doi.org/10.31219/osf.io/fueyj
  • L. Kunold. Seeing is not feeling the touch from a robot, in Proceedings of the 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 1562-1569, September 2022. https://doi.org/10.1109/RO-MAN53752.2022.9900788
  • C. L. Gittens, and D. Garnes, Zenobo on Zoom: Evaluating the human-robot interaction user experience in a videoconferencing session, in 2022 IEEE International Conference on Consumer Electtonics (ICEE), pp. 1-6, March 2022. https://doi.org/10.1109/ICCE53296.2022.9730259
  • M. T. Latkovikj, and M. B. Popovska, Online research about online research: advantages and disadvantages, E-methodology, vol. 6, no. 6, pp. 44-56, May 2020. http://dx.doi.org/10.15503/emet2019.44.56
  • E. Roesler, S. Rudolph, and F. W. Siebert, Exploring the role of sociability, ownership, and affinity for technology in shaping acceptance and intention to use personal assistance robots, International Journal of Social Robotics, February 2024. https://doi.org/10.1007/s12369-024- 01098-1
  • M. Söderlund, Who is who in the age of service robots: The impact of robots' demand for user identification in human-to-robot interactions, Computers in Human Behavior: Artificial Humans, vol. 1, article 100013, September 2023. https://doi.org/10.1016/j.chbah.2023.100013
  • C. S. Arlinghaus, C. Straßmann, and A. Dix, Increased morality through social communication or decision situation worsens the acceptance or robo-advisors, May 2024. http://dx.doi.org/10.31219/osf.io/bufjh
  • C. Straßmann, and N. C. Kramer, A two-study approach to explore the effect of user characteristics on user's perception and evaluation of virtual asisstant's appearance, Multimodal Technologies and Interaction, vol. 2, no. 4, article 66, October 2018. https://doi.org/10.3390/mti2040066
  • K. I. Roumeliotis, and N. D. Tselikas, ChatGPT and Open-AI models: A preliminary review, Future Internet, vol. 15, no. 6, article 192, May 2023. https://doi.org/10.3390/fi15060192
  • R. Chew, J. Bollenbacher, M. Wenger, J. Speer, and A. Kim, LLM-assisted content analysis: Using large language models to support deductive coding, ArXiv, June 2023. https://doi.org/10.48550/arXiv.2306.14924
  • R. H. Tai, L. R. Bentley, X. Xia, J. M. Sitt, S. C. Fankhauser, A. M. Chicas-Mosier, and B. G. Monteith, B. G., An examination of the use of large language models to aid analysis of textual data, International Journal of Qualitative Methods, vol. 23, pp. 1-14, January 2024. https://doi.org/10.1177/16094069241231168
  • C. S. Arlinghaus, C. Wulff, and G. W. Maier, Inductive CodingwithChatGPT - An Evaluation of Different GPT Models Clustering Qualitative Data into Categories, OSF Preprints, July 2024. https://doi.org/10.31219/osf.io/gpnye
  • C. S. Arlinghaus, and G. W. Maier, Clustering statements with ChatGPT – Study 1, Open Science Framework, May 2024. https://doi.org/10.17605/OSF.IO/TQNFK
  • C. S. Arlinghaus, and G. W. Maier, Clustering statements with ChatGPT – Study 2, Open Science Framework, May 2024. https://doi.org/10.17605/OSF.IO/TP4BH
  • SAIL, Sustainable life-cycle of intelligent socio-technical systems, Project website. https://www.sail.nrw
  • S. K. Ötting, L. Masjutin, J. J. Steil, and G. W. Maier, Let's work together: A meta-analysis on robot design features that enable successful human-robot interaction at work, Human Factors, vol. 64, no. 6, pp. 1027–1050, September 2022. https://doi.org/10.1177/0018720820966433
  • K. I. Paul, H. Scholl, K. Moser, A. Zechmann, and B. Batinic, Employment status, psychological needs, and mental health: Meta-analytic findings concerning the latent deprivation model, Frontiers in Psychology, vol. 14, article 1017358, March 2023. https://doi.org/10.3389/fpsyg.2023.1017358
  • K. Isaksson, Unemployment, mental health and the psychological functions of work in male welfare clients in Stockholm, Scandinavian Journal of Social Medicine, vol. 17, no. 2, pp. 165–169, June 1989. https://doi.org/10.1177/140349488901700207
  • C. S. Arlinghaus, and G. W. Maier, Different forms of social exclusion in a robo-restaurant, Open Science Framework, January 2024. https://doi.org/10.17605/OSF.IO/ZAM24
  • OpenAI, Pricing, OpenAI, 2024. https://openai.com/api/pricing/
  • J. Davis, L. Van Bulck, B. N. Durieux, and C. Lindvall, The temperature feature of ChatGPT: Modifying creativity for clinical research, JMIR Human Factors, vol. 11, article e53559, March 2024. http://dx.doi.org/10.2196/53559
  • I. Korstjens, and A. Moser, Series: Practical guidance to qualitative research. Part 4: Trustworthiness and publishing, European Journal of General Practice, vol. 24, no. 1, pp. 120-124, 2018. https://doi.org/10.1080/13814788.2017.1375092
  • J. G. Meyer, R. J. Urbanowicz, P. C. N. Martin, K. O'Conner, R. Li, P.-C. Peng, T. J. Bright, N. Tatonetti, K. Jae Won, G. GonzalesHernandez, and J. H. Moore, ChatGPT and large language models in academia: Opportunities and challenges, BioData Mining, vol. 16, article 20, 203. https://doi.org/10.1186/s13040-023-00339-9
  • S. A. Khowaja, P. Khuwaja, K. Dev, W. Wang, W., and L. Nkenyereye, ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) evaluation: A review, Cognitive Computing, 2024. https://doi.org/10.1007/s12559-024-10285-1