Traditional fault diagnosis method based on Relevance vector machine (RVM) only use a single kernel function or use linearly mixed kernel function. However, these kernel functions have unsatisfactory diagnosis accuracy when the sample data have a complex and high-dimension space distribution. For enhancing the diagnosis accuracy, this paper proposes the RVM with a new nonlinearly mixed kernel function. This kernel function consists of a weighted Gaussian kernel and a weighted polynomial kernel. For searching the optimal weights of this new kernel rapidly, this paper constructs an ant colony system which is similar to the Traveling Salesman Problem (TSP) to search the optimal solution. Finally, this paper uses a real DC power supply switching circuit to verify the high diagnosis accuracy of the ACO-RVM with a new nonlinearly mixed kernel function.