To address the problem of image classification in which the intra-class variation and inter-class similarity are large due to external interference, thus increasing the difficulty of image classification, a lightweight convolutional network Gabor-DCTnet based on Gabor and DCT filters is proposed in this paper. Gabor-DCTnet cascades predefined filter sets FB Gabor-DCT , nonlinear and average pooling operations to make intra-class deformation and inter-class similarity more stable. First, a complex filter FB Gabor-DCT consisting of Gabor filters and DCT bases is constructed in the filtering layer to extract rich image features; second, the nonlinear layer performs binarized hash coding and chunked histogram operations on the filter coefficients to obtain robustness in illumination, rotation, and occlusion; finally, the chunked histogram is averaged pooling and wPCA dimensionality reduction in the pooling layer and input to the nearest neighbor classifier based on cosine distance. The experimental results show that the proposed network can effectively classify images on FERET_I, FERET_II, AR and KTH_TIPS, CUReT datasets, and the classification results are significantly better than those of lightweight networks such as PCANet, DCTNet and M-FFC.