Domain adaptation is widely used in the field of remote sensing, which can transfer the existing knowledge to new tasks and promote performance. When applied in the field of scene classification, it can be called cross-scene classification. Previous cross-scene classification methods mainly consider the coarse-grained alignment in the global aspect, which may ignore the structures behind the data and lose the local information with respect to specific categories. To implement fine-grained alignment, we present an adversarial fine-grained adaptation network (AFGAN) which simultaneously captures the complex structures behind the data distributions to improve the discriminability and reduce the local discrepancy of different domains to align the relevant category distributions. Experimental results based on three existing scene classification datasets demonstrate the effectiveness of AFGAN.