case_images: PMC11346639_01
case_images: Rank-Order stability with PVQ.. Rank-order stability (Mean+-SE) of personal values (PVQ) exhibited by simulated participants (fictional characters or real-world personas) following conversations on different topics (correlation of simulated participants' value expression in different contexts). Consistent trends are visible: Mixtral, Qwen, GPT-3.5, and Mistral model families are the most stable, compared to LLaMa-2 and Phi families. All models exhibit lower than human stability, despite the comparison being skewed in their favor. LLMs are simulating two populations: (A) fictional characters, and (B) real-world personas. For statistical tests, refer to S1 and S2 Figs, respectively.
case_images: pone.0309114.g004.jpg
case_images: PMC11346639_01_pone.0309114.g004.jpg
case_images: pone.0309114.g004
case_images: Vecchione et al. followed 20-year-olds for eight years and Vecchione et al. followed 10-year-old for 2 years (these changes are denoted by horizontal lines in Figs 4 and 5).  Fig 4 shows that all models, when instructed to simulate various personas, exhibit much lower Rank-order stability than that observed in human populations (r = 0.57 for ages 10 to 12, and r = 0.66 for ages 20 to 28). Due to computational constraints (evaluating a model on one population requires 300k queries), we conduct this experiment only with Mixtral-8x7B-Instruct-v0.1 (the most stable model from Fig 4), only on one population (fictional characters) and only with one seed. The model trends are somewhat consistent with the results on PVQ (Fig 4). The trends of model families are consistent with the results on PVQ (Fig 4). the trends of model families are consistent: Mixtral, Mistral, GPT-3.5 and Qwen are again the most stable, while Phi and LLaMa-2 show low stability. These results are consistent with those in previous experiments with five contexts (Fig 4). The following analysis will be made with respect to Rank-Order stability on PVQ (Fig 4, on downstream tasks (Fig 8), and with respect to Ipsative stability of PVQ (Fig 5). Across all experiments (Figs 4, 5, 8 and 9), we observe a slight but consistent drop in stability as a consequence of this quantization.
case_images: Ipsative stability with PVQ.. Ipsative stability (Mean+-SE) of personal values (PVQ) exhibited by LLMs without the persona setting instructions (correlation of value hierarchies in different contexts). Mistral-7B-Instruct-v0.1 and Qwen-72B models show the highest stability. Mixtral, Mistral, GPT-3.5 and Qwen families are more stable. Human change is shown for reference, but no strong conclusions can be made because the comparison is skewed in the LLMs' favor. (Refer to Supporting Information S3 Fig for statistical tests).
case_images: pone.0309114.g005.jpg
case_images: PMC11346639_01_pone.0309114.g005.jpg
case_images: pone.0309114.g005
case_images: Vecchione et al. followed 20-year-olds for eight years and Vecchione et al. followed 10-year-old for 2 years (these changes are denoted by horizontal lines in Figs 4 and 5).  Fig 5 shows the Ipsative stability of models that were not instructed to simulate a persona. We compare the most stable models from Fig 5 without persona instructions, and Mixtral-8x7B-Instruct-v0.1 with instructions to simulate fictional characters. The following analysis will be made with respect to Rank-Order stability on PVQ (Fig 4, on downstream tasks (Fig 8), and with respect to Ipsative stability of PVQ (Fig 5). Furthermore, despite large differences in size, all LLaMa-2 models consistently exhibit low stability, with the exception of the Ipsative stability of LLaMa-2-70B-chat (Fig 5) and a modest Rank-Order stability on the Religion downstream task of both LLaMa-2-70B models (Fig 8C). Across all experiments (Figs 4, 5, 8 and 9), we observe a slight but consistent drop in stability as a consequence of this quantization.
case_images: Rank-Order stability with longer conversations.. Rank-order value stability (Mean+-SE) following conversations of different length for the Mixtral-8x7B-Instruct-v0.1 model simulating fictional characters (correlation of simulated participants' value expression in different contexts). Stability decreases with longer simulated conversations. For statistical tests, refer to S4 Fig.
case_images: pone.0309114.g006.jpg
case_images: PMC11346639_01_pone.0309114.g006.jpg
case_images: pone.0309114.g006
case_images:  Fig 6 shows the effect of simulated conversation length on Rank-order stability expressed by the Mixtral-8x7B-Instruct-v0.1 model instructed to simulate fictional characters.
case_images: Ipsative stability with longer conversations.. Ipsative value stability (Mean+-SE) of LLMs with (Mixtral-8x7B-Instruct-v0.1) and without persona setting instructions (correlation of value hierarchies in different contexts). All models retain the same stability level in longer conversations.
case_images: pone.0309114.g007.jpg
case_images: PMC11346639_01_pone.0309114.g007.jpg
case_images: pone.0309114.g007
case_images:  Fig 7 shows the effect of conversation length on Ipsative stability. Ipsative stability (Fig 7). Ipsative stability even increases following a very long conversation with 43 simulated messages.
case_images: Rank-Order stability on downstream tasks.. Rank-order stability (Mean+-SE) on downstream tasks of various LLMs (correlation of simulated participants' behavior in different contexts). Three downstream tasks are shown: (A) Donation, (B) Stealing, and (C) Religion. For statistical test, refer to S4-S6 Figs, respectively. Consistent trends with the PVQ experiments (Fig 4) are visible. Mixtral, Qwen, GPT-3.5, and Mistral model families are the most stable, compared to LLaMa-2 and Phi families. Mixtral-8x7B-Instruct-v0.1, Mistral-7B-Instruct-v0.2, gpt-3.5-turbo-1106 and Qwen-72B are the most stable models. Trends are the most present on the, easiest, Religion task (c) and almost disappear on the, hardest, Stealing task (b).
case_images: pone.0309114.g008.jpg
case_images: PMC11346639_01_pone.0309114.g008.jpg
case_images: pone.0309114.g008
case_images:  Fig 8 compares models' stability on the three downstream tasks. In comparing the overall stability levels, the Stealing task appears to be the hardest (Fig 8B), followed by the Donation (Fig 8A) task and the Religion task (Fig 8C). The following analysis will be made with respect to Rank-Order stability on PVQ (Fig 4, on downstream tasks (Fig 8), and with respect to Ipsative stability of PVQ (Fig 5). Furthermore, despite large differences in size, all LLaMa-2 models consistently exhibit low stability, with the exception of the Ipsative stability of LLaMa-2-70B-chat (Fig 5) and a modest Rank-Order stability on the Religion downstream task of both LLaMa-2-70B models (Fig 8C). Across all experiments (Figs 4, 5, 8 and 9), we observe a slight but consistent drop in stability as a consequence of this quantization.
case_images: Relation of value expression on PVQ and donating behavior.. Rank-order stability (Mean+-SE) between value expression (on the PVQ questionnaire) and the donation amount (correlation between simulated participants' value expression and donation behavior). For more stable modes, donations are correlated with Universalism (a) and Benevolence (b) and negatively correlated with Power (c) and Achievement (d).
case_images: pone.0309114.g009.jpg
case_images: PMC11346639_01_pone.0309114.g009.jpg
case_images: pone.0309114.g009
case_images:  Fig 9 shows the correlation between rank-order of value expression on PVQ and the donation amount on a downstream task. Across all experiments (Figs 4, 5, 8 and 9), we observe a slight but consistent drop in stability as a consequence of this quantization.
case_images: Rank-Order stability on additional contexts.. Pair-wise Rank-Order stability of personal values (PVQ) exhibited by simulated fictional characters. The Mixtral-Instruct-8x7B-v0.1 model overall exhibited higher stability than Mistral-7B-Instruct-v0.2. For both models, lower stability is observed in longer contexts (bottom right corner).
case_images: pone.0309114.g010.jpg
case_images: PMC11346639_01_pone.0309114.g010.jpg
case_images: pone.0309114.g010
case_images:  Fig 10 shows stability between each pair of contexts for Mistral-7B-Instruct-v0.2 and Mixtral-8x7B-Instruct-v0.1. Furthermore, the contexts in Fig 10 are ordered based on the length of the initial message.