Ghost imaging (GI), as a new computational imaging method, has a great prospect of development. However, speckles that interact with objects may not be consistent with the measured due to interference in the propagation path. In recent years, deep learning (DL) methods have been more and more applied to GI. However, most neural networks are sensitive to speckle patterns change, which affects the reconstruction performance. In order to avoid the impact of patterns change on network performance, we here proposed a network adaptation method for GI patterns change. The experimental results show that the reconstruction performance of our method is similar to that without patterns change.