The domain adaption (DA) based on deep learning has achieved great success in cross domain unsupervised fault diagnosis. In general, it assumes that the label sets of training data (source domain) and test data (target domain) are consistent. However, in practice, the test data usually include unknown samples that are not observed in the training data due to unpredictable fault modes in the test phase. Therefore, the open set fault diagnosis (OSFD) algorithm where the label set of training data is a part of the label set of test data came into being. OSFD aims to identify unknown fault modes in the target domain and align the source domain and target domain to identify known fault modes. However, most previous studies directly align the source domain and target domain without considering the unique information of each class, which will make the decision boundary of the classifier indivisible. In this paper, we expand the decision boundary for OSFD from two aspects. First, the decision boundary between the unknown samples and all known classes is expanded, i.e., the unknown samples are pushed away from the decision boundary. Then, the decision boundary between all known classes is expanded, i.e., to align the known classes more accurately. On the twin-spool engine experiment, our method can precisely identify unknow fault modes and accurately classify known fault modes compared with previous methods, and the effectiveness of the method was verified by feature visualization and ablation study.