The development of Artificial Intelligence (AI) systems in Industry 4.0 has gained momentum due to their potential for increasing efficiency and productivity. However, AI systems can be just as complex and opaque, leading to concerns about their reliability, trustworthiness, and accountability. To address these issues, this paper proposes a validation framework for Explainable AI (XAI) in Industry 4.0 based on Fuzzy Cognitive Maps (FCMs). The proposed framework aims to evaluate Key Performance Indicators (KPIs) based on a set of AI metrics and XAI metrics. The FCM-based approach enables the representation of causal-effect relationships between the different concepts of the system using expert knowledge. The presented validation framework provides a theoretical background for evaluating and optimizing the business values of AI systems based on multiple criteria in the manufacturing industry, demonstrating its effectiveness. The main contributions of this paper are: i) the development of an FCM-based validation framework for XAI in Industry 4.0; ii) the identification of relevant AI and XAI metrics for the evaluation of the KPIs of the theoretical graph model; and iii) the demonstration of the effectiveness of the proposed framework through a case study. The results of this study provide valuable insights into the importance of considering not only accuracy but also efficiency and transparency when developing AI pipelines that generate higher business value. Overall, this paper offers a theoretical foundation and practical insights for organizations seeking to evaluate the business values of their AI systems in Industry 4.0. It emphasizes the importance of explainability and the integration of AI and XAI metrics in achieving transparent and accountable AI solutions that deliver optimal results for the manufacturing industry and beyond.