The surface features around railway can provide necessary environment information for railway survey and design, construction management, operation and maintenance, which provide information support for railway design and route selection, land demolition and land acquisition, and foreign object intrusion restrictions. However, the traditional artificial method has problems such as large manpower occupation, long time consumption and high detection missed rate. Aiming at the above problems, we proposed the feature cross fusion-attention mechanism for intelligent semantic segmentation of surface features around railway. On the basis of the hyperseg, the features of adjacent layers are fused by cross-convolution, which improves the information sharing between features of different scales. The attention module is constructed between the same layers of upsampling and downsampling. The local features in the multi-scale fusion feature map are weighted by the attention module and fused with the upsampling features to enrich the detailed local features in the upsampling feature maps and improve the accuracy of target segmentation. The experimental results show that the segmentation accuracy of the color steel houses and buildings of proposed method can reach 0.9235 and 0.7921, and the mIOU can reach 0. S761 and 0.6852.