Fraud detection is relevant in this context because of the large scale of online transactions, which conceal a small number of fraudulent transactions.Machine learning (ML) fraud detection models have numerous hyperparameters of the eXtreme Gradient Boosting (XGBoost) algorithm, which are important for the performance impact of fraud detection models. In this paper, the propose an improved version based on Sailfish optimizer(SFO) applied to online transaction fraud detection to optimize the hyperparameters of the XGBoost algorithm. Firstly, the optimization-seeking and local-optimal-skipping capabilities of Sailfish optimizer are improved, and Improved Sailfish optimizer (ISFO) is compared with other classical optimization algorithms, and the results show that the algorithm has outstanding. The Improved Sailfish optimizer is then used to optimize the parameters of XGBoost (ISFO-XGBoost) and compared with several other machine learning classifiers such as: Decision Tree (DT), Random Forest (RF), XGBoost, and several optimization algorithms optimized for the XGBoost model. To validate the performance, a fraud dataset of online transactions is used for evaluation. The experimental results demonstrate that the proposed improved algorithm is highly promising in solving machine learning hyperparameter optimization challenges, and the ISFO-XGBoost algorithm achieves the best results.