Currently, clinical methods for Parkinson's disease (PD) diagnosis are not very effective, and there is an urgent need for a more accurate diagnostic approach. When using Magnetic Resonance Imaging (MRI) for PD diagnosis, relying solely on T1-weighted (T1) or Quantitative Susceptibility Mapping (QSM) images cannot comprehensively consider the information about different types of brain lesions, leading to a bottleneck in improving the classification accuracy. This study utilize a 3D convolutional neural network (3D-CNN) to integrate multi-modal MRI data for PD diagnosis. Clinical data shows accuracy of multi-modal 3D-CNN is higher than accuracy of single-modal 3D-CNN, when all tissue data is used as input, the classification accuracy of multi-modal 3D-CNN can reach 91%, which can prove that the multi-modal fusion method is superior to the direct use of single-modal data. Furthermore, by focusing on several basal ganglia structures that were reported to be significantly affected by PD, this 3D-CNN model shows that substantia nigra, caudate, and especially thalamus have a high sensitivity for PD diagnosis. The findings indicate the potential of using multi-modal deep learning approaches for PD diagnosis and suggest that the selected basal ganglia structures are highly sensitive markers for PD detection. These results offer promising prospects for the development of more effective and accurate PD diagnostic methods.