Arrhythmia indicates abnormal sources of electrical impulses in the heart or abnormalities in any conductive process. The electrocardiogram (ECG) is the most popular tool for detecting arrhythmia as it is non-invasive and effective. As visual inspection is difficult and tedious, computer-aided diagnosis (CAD) is developed to provide medical decision supports. In this study, a method based on convolutional neural network (CNN) is proposed to automated classify heartbeat of arrhythmia. Compared with conventional machine learning methods, CNN requires no additional feature extraction or feature selection processes. The ECG signal is segmented and then the heartbeats are inputted to the model directly. The structure of CNN in this study is improved. In one convolution layer, kernels with different sizes are used simultaneously. Afterward, the max-pooling operation is applied to the feature maps. Finally, the feature maps generated by kernels with different sizes are concatenated and inputted to the fully-connected layers. The method was employed on the MIT-BIH arrhythmia database and an accuracy of 98.73% was obtained. A comparative experiment was conducted to demonstrate the effectiveness of the proposed method.