Based on the development of ghost imaging and deep learning, this letter proposes a double encryption method of ghost imaging by SegNet deep neural network by fully combining the advantages of both. In the sender, the secret image and host image are encrypted by hiding network structure and ghost imaging system to complete the first level and the second level encryption to obtain the ciphertext. In this encryption method, a hiding network is constructed based on SegNet, and important information is preserved by concatenation convolution of two images, and image steganography is realized by encoding. The revealed network based on CNN is constructed to realize the secret image extraction. Through the analysis of the encryption results, can be seen that this encryption scheme has higher efficiency and better effect in information encryption compared with other image encryption methods, and has more practical research significance and application value.