In general, cover source mismatch (CSM) inevitably leads to a significant decrease in detection accuracy in image steganalysis because the source and target domains have different distributions. To remedy this problem, a universal mismatched steganalyzer ISNet equipped with generated local mixing positions, a local feature-level mixup-related patchup (LFMP), and domain factors is proposed for both spatial and JPEG domains in this paper. Unlike existing deep steganalysis networks, which simply minimize the domain discrepancy between source and target to address the problem of CSM, and thus, cannot handle large domain discrepancy, IDGM based on LFMP generates diverse intermediate domains to bridge the two extreme domains so as to alleviate the decrease in detection accuracy caused by CSM. Moreover, ISNet enables the intermediate domain distribution to progressively transit from source to target by adjusting the domain factor sampled from $Beta(\alpha,1)$, so that the classifier gradually adapted to target provides an improvement in discriminability on the target domain. The experimental results show that ISNet achieves the best performance in various CSM cases, compared with the most advanced deep learning-based steganalysis network.