Speaking ability is widely recognized as a key element of learning a foreign language. Since the emergence of the Common European Framework of Reference for Languages (CEFR), many studies have explicated the relationship between second language learners’ speaking performance and CEFR levels. To provide this issue with strong evidence, large-scale language test data may considerably impact the field of language teaching and assessment. Thus, the present study investigated the statistical differences of five aspects of speaking performance (i.e., complexity, accuracy, fluency, interactional effectiveness, and the amount of talk) between four CEFR levels (Below A2, A2, B1, and B2) based on the performance of the tasks in the speaking section of the Test of English for Academic Purposes (TEAP). The data analyzed were of the speaking performance of 153 high school students in Japan. The results suggest that the variables of the amount of talk (i.e., the number of words in a long turn, the number of words in all parts, and the number of AS units) and fluency (i.e., words per minute, syllables per second, and the ratio of pause time to speaking time in a one-minute speech task) differed between the four CEFR levels. The other measures also represent the differences in speaking performance at different adjacent levels. Based on the findings of this study, implications for L2 teaching and assessment of L2 performance are discussed.