The fuzzy c-means algorithm is a popular choice in clustering data sets with soft boundaries. In the last few decades, FCM has widely been used in the different fields of science and technology, especially for the purpose of image processing, medical diagnosis, pattern recognition, etc. However, the performance of fuzzy c-means clustering deteriorates with higher dimensions, in the presence of noise and due to high initial bias. The article coins a new approach to improve the performance of the conventional FCM based on the ensemble method. Although there are some approaches using the ensemble method, the proposed algorithm differs from these approaches in the base learner used. The proposed algorithm uses the concept of supervised learning in clustering to generate the base learners and then combine them to find the final cluster solution. In this method, the given data set is segregated into more confident data points and less confident data points. The more confident data points are further clustered using the k-means algorithm. The cluster labels of the less confident data points are estimated by a k-nearest neighbor classifier from the cluster labels of the more confident data points. The performance of our proposed method is checked on two medical data sets viz., thyroid data set and MRI brain image data set. Three performance measures namely the rand index, adjusted rand index, and Minkowski score are used to compare our proposed algorithm with the conventional fuzzy c-means clustering algorithm. The simulation results show the efficacy of our proposed algorithm over the individual runs of FCM. [ABSTRACT FROM AUTHOR]