The aim of this paper is to analyze dynamic characteristics and stability of magnetic sail-based planetary displaced orbits, and the feasibility of using a magnetic sail as an advanced propellantless control technology for formation flying around elliptic planetary displaced orbits (EPDOs). The thrust can be obtained from the momentum exchange between solar wind and an artificial magnetic field. First, the requirements of a magnetic sail for generating and maintaining a planetary circular displaced orbit is discussed including the value of attitude angles and characteristic acceleration. Based on different orbital periods, the circular displaced orbits are divided into three types and their linear stability is analyzed. For elliptical orbits, the conditions required to maintain an EPDO are investigated and the dynamical models of magnetic sail formation system are established. A finite time coordinated control algorithms relying on the protocols formulated on an undirected communication graph is proposed to achieve synchronized formation tracking rapidly while enhancing the robustness of formation system due to information interaction between spacecraft. Several numerical simulations are conducted to demonstrate that spacecraft formation can be effectively controlled by the proposed propellantless propulsion system.