Despite recent advancements in AI and Data Science, the vast Big Data sources available to medical and health care providers are far from living up to their potential. Addressing the underlying transparency and data protection concerns helps to unleash these advancements in the medical domain and thus benefits research and patient care. Subsequently, we propose a system for trustworthy medical operational AI in this paper. We present our vision to align medical operational AI with regulatory demands of the medical domain. We propose guiding principles to marry data-driven diagnostic recommendations with legal frameworks, clinical protocols, and expert-driven reasoning. Through this research, we aim for AI systems in medicine that not only provide accurate predictions but also empower users to comprehend and trust the underlying decision-making processes.