In this paper, an adaptive radial basis function neural network (RBFNN) controller based on extraordinariness particle swarm optimization (EPSO) is proposed. To improve the trajectory tracking performance of robotic manipulators, the uncertainties of the manipulator dynamic equation are locally approximated using three RBFNNs with optimized hyperparameters. Besides, a robust control item is also considered in the controller to resist external disturbances. During hyperparameters optimization, the EPSO optimizer iteratively optimizes the hyperparameters of the RBFNN controller using the composite error of the system output. The stability of the control scheme is analyzed with the Lyapunov stability. Simulation results as well as the experimental verification prove the efficiency and applicability of the control scheme.