Speech has lots of emotional information and speech emotional recognition is one of the top topic in AI field. Traditional research extracted speech feature first and then to classified the emotions by sorting algorithms. However, all kinds of feature extracting algorithms lost some original speech information to some extent, so the accuracy rate is reduced. In this paper, we present an end-to-end speech emotion recognition system which based on neural network without feature extraction. Experimental testing of the proposed scheme was performed using the DMO-DB (German) and CASIA (Chinese) emotional speech datasets, recognition rates reached 74.12% and 44.5% respectively.