Quantitative Susceptibility Mapping (QSM) is a new phase-based technique for quantifying magnetic susceptibility. The existing QSM reconstruction methods generally require complicated pre-processing on high-quality phase data. In this work, we propose to explore a new value of the high-pass filtered phase data generated in susceptibility weighted imaging (SWI), and develop an end-to-end Cross-connected $\Psi$-Net (C$\Psi$-Net) to reconstruct QSM directly from these phase data in SWI without additional pre-processing. C$\Psi$-Net adds an intermediate branch in the classical U-Net to form a $\Psi$-like structure. The specially designed dilated interaction block is embedded in each level of this branch to enlarge the receptive fields for capturing more susceptibility information from a wider spatial range of phase images. Moreover, the crossed connections are utilized between branches to implement a multi-resolution feature fusion scheme, which helps C$\Psi$-Net capture rich contextual information for accurate reconstruction. The experimental results on a human dataset show that C$\Psi$-Net achieves superior performance in our task over other QSM reconstruction algorithms.