To successfully move a robot into the building, the elevator button and elevator floor number detection and recognition can play an important role. It can help a robot move in the building, just as it also can help a visually impaired person who wants to move another floor in the building. Due to vision-based approach, the difference in lighting condition and the complex background are the main obstacles in this research. A hybrid image classification model is presented in this research to overcome all these difficulties. This hybrid model is the combination of histogram of oriented gradients and bag of words models, which later reduces the dimension of image features by using the feature selection algorithm. An artificial neural network has been implemented to get the experimental result by training and testing. In order to get training performance, 1000 training image samples have been used and additional 1000 image samples also been used to get the testing performance. The experimental results of this research indicate that this proposed framework is important for real-time implementation to implement the elevator button and elevator floor number recognition framework.