Coffee is an important economic crop and one of the most popular beverages worldwide. The rise of specialty coffees has changed peoples standards regarding coffee quality. However, green coffee beans are often mixed with impurities and unpleasant beans. Therefore, this study aimed to solve the problem of time-consuming and labor-intensive manual selection of coffee beans for specialty coffee products. The second objective of our study was to develop an automatic coffee bean picking system. We first used image processing and data augmentation technologies to deal with the data. We then used deep learning of the convolutional neural network to analyze the image information. Finally, we applied the training model to connect a webcam for video streaming recognition. We successfully divided the good and bad beans. The false positive rate was 0.1007, and the overall coffee bean recognition rate was 93%.