Generating molecular graphs using deep graph generative models is a challenging task that involves optimizing a given target within an enormous search space while adhering to chemical valence rules. Despite promising results, existing models mainly focus on learning molecular graph structures at the individual level while ignoring inter-molecular relationships regarding molecular characterization features and molecular activity. This can lead to the generation of molecules that are unresponsive to their true neighbors possessing similar characterization features, resulting in a divergence between the learned generation distribution and the actual molecular distribution. In this paper, we propose a distribution preserving model, designed to maintain the inter-molecular relationships of the original distribution within the generated space. Specifically, the model operates on a student-teacher paradigm, where the teacher module learns the inter-molecular relationship dynamics of the original distribution, and imparts this knowledge to the student module, which is responsible for generating molecules. Experimental results on two benchmark datasets demonstrate that our model outperforms the state-of-the-art models in generating valid, novel and unique molecules. Moreover, our model is verified on preserving molecule distribution in the generation space.