Agentic AI for Real-Time, Resilient, and Adaptive Fraud Detection in Digital Payment Systems

With the rapid rise of digital payment systems, the risk of fraudulent transactions has become a pressing concern, underscoring the need for intelligent, real-time, and adaptive detection mechanisms. This study leverages Agentic AI, a goal-driven and autonomous paradigm, to design an end-to-end fraud detection framework that integrates ensemble learning, contextual reasoning, and self-optimization. The proposed architecture combines Graph Neural Networks (GNNs) for relationship-aware transaction modeling with Transformer-based anomaly detection to capture temporal–sequential irregularities. An autonomous policy engine drives adaptive thresholding, enabling the system to dynamically respond to changing fraud patterns. To address the severe data imbalance—where fraudulent cases represent only 0.15\% of the dataset—the study introduces a Dynamic Synthetic Oversampling with Reinforcement Feedback (DSORF) mechanism. DSORF iteratively generates synthetic fraud samples guided by model feedback, ensuring improved detection while maintaining dataset integrity. Experimental results highlight the framework's effectiveness, achieving 99.96% accuracy, 91.84% precision, and 89.12% recall, while significantly reducing false positives compared to state-of-the-art static methods. These findings demonstrate the potential of Agentic AI to autonomously adapt to evolving fraud landscapes, offering enhanced resilience, scalability, and reliability for digital payment systems. Looking forward, future research will focus on extending the framework to cross-border and multi-currency payment scenarios while incorporating real-time decision explainability, further strengthening trust and transparency in AI-driven financial security.