Fuzzing Inference-Time Privacy Risks in Deployed Large Language Models

Large Language Models (LLMs) are increasingly deployed in applications that process sensitive information, raising concerns about privacy violations during inference. While existing approaches primarily focus on training-time guarantees and static defenses, they provide limited insight into whether deployed models exhibit privacy-violating behavior at runtime. In this paper, we propose a fuzzing-based testing framework for evaluating inference-time privacy risks in LLMs under black-box conditions. The framework operationalizes privacy risks as executable artifacts composed of structured test seeds, risk-specific prompt transformations, and explicit test oracles. We instantiate the framework for the misuse and malicious use of personal data risk using canary injection and multi-message conversational trajectories. An empirical evaluation across three commercial LLMs demonstrates that privacy failures can be systematically elicited, are highly dependent on conversational structure and identifier representation, and vary substantially across models. The results indicate that privacy can be treated as a testable behavioral property at inference time, enabling comparative and risk-driven evaluation of deployed LLMs.