Facial expression recognition and gender classification has many applications in affective computing as well as computer vision respectively. The main applications involve human computer interaction, driver safety etc. Principal component analysis(PCA), Linear discriminant analysis (LDA), Linear binary pattern(LBP) algorithms are used in most cases for the detection of facial expression and gender. The drawbacks of existing systems include lower classification rate in the case of low resolution images, confusion between different pairs of expression etc. In this paper, a novel method for facial expression recognition and gender classification based on the two expressions anger and joy along with geometric and appearance based method is proposed. Facial patches are used to detect both gender and facial expression. Facial expression is identified based on the appearance of facial patches. The proposed system successfully tested with Japanese female facial expression (JAFFE) database and Cohn-Kande databases.