Face forgery technologies may have a significant adverse impact on individual privacy and national political security. In this paper, a simple but effective method for detecting forgery face image based on improved capsule network is proposed. More specifically, we first adopt the part of per-trained VGG19 to extract latent features for better classification. Then, the improved capsule network architecture makes use of exponential linear unit (ELU) instead of the traditional rectified linear unit (ReLU) to improve the learning speed and convergence properties. Moreover, the effective attention mechanisms are embedded into the improved capsule network for further improving the detecting accuracy performance. Experimental results on four famous face forgery datasets demonstrate that the proposed framework outperforms other state-of-the-art approaches.