Active Learning of Markov Decision Processes using Baum-Welch algorithm
- Resource Type
- Conference
- Authors
- Bacci, Giovanni; Ingolfsdottir, Anna; Larsen, Kim G.; Reynouard, Raphael
- Source
- 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA) ICMLA Machine Learning and Applications (ICMLA), 2021 20th IEEE International Conference on. :1203-1208 Dec, 2021
- Subject
- Computing and Processing
General Topics for Engineers
Power, Energy and Industry Applications
Robotics and Control Systems
Signal Processing and Analysis
Adaptation models
Machine learning algorithms
Learning automata
Heuristic algorithms
Conferences
Manuals
Machine learning
Baum-Welch algorithm
Markov decision processes
active learning
- Language
Cyber-physical systems (CPSs) are naturally modelled as reactive systems with nondeterministic and probabilistic dynamics. Model-based verification techniques have proved effective in the deployment of safety-critical CPSs. Central for a successful application of such techniques is the construction of an accurate formal model for the system. Manual construction can be a resource-demanding and error-prone process, thus motivating the design of automata learning algorithms to synthesise a system model from observed system behaviours.This paper revisits and adapts the classic Baum-Welch algorithm for learning Markov decision processes and Markov chains. For the case of MDPs, which typically demand more observations, we present a model-based active learning sampling strategy that choses examples which are most informative w.r.t. the current model hypothesis. We empirically compare our approach with state-of-the-art tools and demonstrate that the proposed active learning procedure can significantly reduce the number of observations required to obtain accurate models.