Visual domain adaptation can utilize similar but different source data to solve the scarcity problem of labeled data in target domain, therefore it has received more and more attention. The main difficulty of this task is the distribution discrepancy between target data and source data. Hence, a series of works have been proposed to reduce the distribution discrepancy by seeking cross-domain feature invariant, or reducing the weight of source instances that are not related to the target data. The major disadvantage is that most works fail to consider the intrinsic geometric manifold structure, which is important for effective learning. This paper considers the intrinsic geometric manifold structure, and the obtained domain invariant features can be more discriminating. Our method effectively integrates distribution matching and preserving the structure consistency into one framework, and obtains the optimal solution by solving a generalized eigen-decomposition problem. Comprehensive experimental results verify that proposed method can significantly outperform state-of-the-art methods in cross domain image recognition problems.