Due to communication latency with remote ground sites, automatic recognition of Mars terrain is essential for the path-planning of rovers. Currently, most vision-based terrain classification require thousands of fine-grained training samples, while the undefined terrains on Mars are difficult to be classified or fine-grained labeled. Actually, most of the terrain categories can only be coarse-grained labeled due to several limitations, such as overlapped sub-regions, blurred borders, etc. To solve this problem, CACMT (Coarse-grained Annotation-based Classification for Mars Terrain) is proposed to generate the global fine-grained classification map from the local coarse-grained data. Specifically, the complete pipeline is decomposed into (i) annotation rules with unique design, (ii) hierarchical feature fusion network for predicting sub-features of terrain (iii) and a generator for outputting dense terrain categories of Mars. Finally, the results of actual data on Mars demonstrate that the terrain sub-features can be successfully recognized and a dense terrain classification map can be generated applying only coarse-grained labeled images.