This paper proposes Safe Tracker, an autonomous aerial tracking framework based on vision sensor that can deal with challenging tracking missions and guarantee safety and visibility. Firstly, a RGB-D sensor based segmentation method is employed to locate the target in real-time. Then target motion is predicted for a short time horizon, based on which a heuristic path searching method is applied to generate an occlusion-free path. Finally, particular formulations to balance visibility and safety are designed, and an effective non-linear trajectory optimization method enables to generate an optimal tracking trajectory. Autonomous aerial tracking experiments using a vision sensor are conducted to demonstrate the effectiveness of the proposed Safe Tracker in a challenging cluttered forest and a room full of sharp turns using Gazebo. Benchmark comparisons validate that our method tracks more safely and robustly than the state-of-art method.