The facial expression recognition has become one of the research focuses in recent years. Since smile is one of the most significant facial expressions, the recognition of smile can contribute to the development of the research of human facial expression recognition. In this paper, an automatic system for smile recognition is proposed. Face areas are firstly extracted from the original images. Meanwhile, on the basis of face extraction, the convolutional neural network (CNN) is trained by different optimizers. The experiments show that the highest accuracy of this system is 93.16% by RMSProp with momentum. And the maximum accuracy for face images is 92.09% by Adam optimizer while it's just 69.53% for original images, which suggests the combination of face detection and CNN can further improve the performance of the classifier.