Recognizing Customer Behavior (CB) from videos of instore cameras is important to smart retail solutions. Because of possible changes in retail needs and environments, a high degree of adaptability to different target CBs is required for Customer Behavior Recognition (CBR) methods. Existing CBR methods are mainly machine learning based models due to their remarkable recognition accuracy. However, trained models are not reusable for different target CBs. Consequently, existing CBR methods are hard to adapt to different target CBs because the necessary recollecting data and retraining models. In this paper, we propose a CBR method that recognizes CBs by combinations of primitives, each of which represents an object’s motion or objects’ relationship. Since primitives can be reused in combinations for various CBs, the proposed method is easily adaptable to changed target CBs. Experiments on two datasets indicate the good adaptability and sufficient recognition accuracy of our method.