In this paper, the authors present a new architectural concept named as Internet-of-Explainable-Digital-Twins (IoxDT), that holistically integrates explainable AI (xAI) and digital twin (DT) in Industry 5.0 production processes. The authors present the architecture over a versatile production system (VPS) corn production plant, where processes are sensor-driven and monitored to minimize bias. The collaboration of DT and xAI leverages the robustness, predictability, and maintainability of VPS processes. In the proposed architecture, the VPS sensor-controlled inputs are sent to the DT controller. The controller is supplied with virtual inputs, which are replicas of real inputs, and the controller behaviour is monitored. The outputs are connected to an xAI module designed with Shapley additive explanations (SHAP) model, which generates the output predictability and dependence on inputs. The unsafe outputs are rejected. A case-study evaluation is carried out in terms of the mean and standard deviation of DT module creation, and SHAP predictability outputs, which shows an accuracy of 96% of unsafe inputs. The proposed results indicate the viability of the framework in real Industry 5.0 VPS setups.