arxiv_supplementary_material_User_study_dataset

**Supplemental Material Description for "Expanding Horizons in HCI Research Through LLM-Driven Qualitative Analysis"**

The Excel workbook titled `arxiv_supplymentary_material_User_study_dataset.xlsx` provided as supplementary material holds the dataset necessary for understanding the evaluation methods used in the study. The contents are organized into several sheets, described as follows:

1. **Paper**: In this sheet, the research paper selection is listed, including an identifier (ID), the title of the paper, its publication year, the venue where it was presented, the DOI, and information regarding the availability of raw data. This dataset only includes papers that offer open access to raw participant data for qualitative analysis, foundational to the research process.

2. **GPT_paper_summary**: This sheet contains summaries of the selected papers, created using GPT-4. Included for each paper are the ID, the title, and the GPT-4-generated summary, which were required by the system for initializing the analysis.

3. **G1, G2, … , G5**: Recorded in these tabs are the results from multiple rounds of applying the evaluation system. Each 'Gx' tab corresponds to one trial, and contains both the paper ID as well as the qualitative comments crafted by the LLM, mirroring the deductive reasoning typically manifest in human analysis.

It is important to note that the data shared excludes any material that is not cleared for public release, including full-text articles and the raw responses of the study participants. The dataset, therefore, aligns with copyright and privacy policies while providing an adequate basis for verification and exploration by other researchers.

The LLM-generated discussions are compared with the original papers' discussions to assess the capabilities of LLMs in reconstructing meaningful narrative in the absence of the source material. This dataset supports the credibility of the study's outcomes and presents an opportunity for others in the field to undertake similar research endeavors.