ChatGPT, as a recently launched large language model (LLM), has shown superior performance in various natural language processing (NLP) tasks. However, two major limitations hinder its potential applications: 1) the inflexibility of finetuning on downstream tasks, and 2) the lack of interpretability in the decision-making process. To tackle these limitations, we propose a novel framework that leverages the power of ChatGPT for specific tasks, such as text classification, while improving its interpretability. The proposed framework conducts a knowledge graph extraction task to extract refined and structural knowledge from the raw data using ChatGPT. The rich knowledge is then converted into a graph, which is further used to train an interpretable linear classifier to make predictions. To evaluate the effectiveness of our proposed method, we conduct experiments on four benchmark datasets. The results demonstrate that our method can significantly improve the prediction performance compared to directly utilizing ChatGPT for text classification tasks. Furthermore, our method provides a more transparent decision-making process compared with previous text classification methods. The code is available at https://github.com/sycny/ChatGraph.