Robotics has been widely used in our daily life in recent years. However, programming and management for heterogeneous robot swarms are still very challenging. In this paper, we present an Actor-based programming framework for simultaneous management of multiple heterogeneous robot swarms. Our framework encapsulates robotic behavior as the ‘Actor’ by using the OODA (observation, orientation, decision, action) bus-plugin mechanism, and takes ‘Actor’ as the basic control unit. Based on the proposed ‘Actor’, we further designed a domain-specific language (DSL) to decouple the development into task development and algorithm development. Besides, we implement a novel communication middleware and the backbone of a multi-layer Actor model to support heterogeneous robot swarms. We implement our proposed framework in C++ and evaluate it on heterogeneous swarm platforms. The experimental results show that our framework can support the development and management of heterogeneous robot swarms.