Ultra-fast oxide traps with a response time of less than 100 ns can be critical for DRAM operation. However, no investigation can be carried out due to the lack of a characterization method. In this paper, we use the time-voltage-swept charge pumping technique and successfully characterized these ultra-fast traps. Considering the non-radiative multi-phonon transition for trapping/de-trapping processes, we accurately obtained their spatial and energy distribution. In comparison with ab-initio calculations, it is revealed that these ultra-fast traps are related to nitrogen substitution defects, possibly introduced during processes like decoupled plasma nitridation.