Image matching is the critical technique for heterogeneous information fusion in the big data era. To solve the nonlinear radiation distortion problem between optical and synthetic aperture radar (SAR) images, an adaptive multi-scale matching method (AMM) was proposed. First, the edge features of two images were extracted in combination according to their imaging mechanisms. Second, core point matching combining phase features was applied to adaptively determine the approximate offset between two image centers. Third, matching points were obtained by wavelet multi-scale matching. Finally, the geometric relationship of matched images was determined after the outlier removal using the fast sample consensus (FSC) method. The matching performance of the proposed method was quantitatively compared with that of histogram of orientated phase congruency (HOPC) and mutual information (MI). The experiment shows that the proposed method is superior to HOPC and MI in terms of matching accuracy and matching efficiency.