Port operation vehicles are mainly responsible for the transshipment of goods. If there is inadequate supervision in the process of transshipment it is easy to cause such behaviors as cargo leakage, cargo theft and illegal parking of drivers, which cause economic losses to the port. In order to discover such behaviors in time, the unsupervised anomaly detection algorithm Self-encoder-based Deep Feature Fusion Model(S-DFFM) is proposed to judge whether the trajectory of operation vehicles is abnormal or not. The method comprehensively characterizes the trajectory by fusing the shallow features of low-dimensional trajectory and the deep features of high-dimensional trajectory, which frees the trajectory from the limitation of spatial attributes. The experimental data adopts the real trajectory data of one month (7,547 trips) of operating vehicle trajectories of a port in Chongqing, and the experimental results show that S-DFFM can better represent the trajectory features, and the accuracy of trajectory abnormality detection using S-DFFM is as high as 96%.