In the electroencephalogram (EEG)-based emotion recognition task, although the multichannel acquisition method has advantages, it brings operational difficulty and increases the mental stress of the subjects, which affects the data quality. To effectively mine the emotional information in EEG and select the optimal critical channel, we propose a novel critical channel selection framework for EEG emotion recognition: TFBOC-WOHDNN, and conduct extensive experiments on the SEED dataset. First, using the brain function network topological feature as a new independent evaluation index is proposed to filter 30 suboptimal critical channels in four brain regions. To effectively model the spatiotemporal dependence of EEG signals, a hybrid deep model WOHDNN is proposed that can automatically find the optimal hyperparameters. On this basis, an improved binary optimization channel selection method is proposed to effectively screen the optimal critical channels for a specified number of channels. We found that high-band gamma and beta contribute more to EEG emotion recognition, and the temporal lobe and frontal lobe are critical brain regions. The selected 4-channel, 8-channel, and 12-channel optimal critical channel schemes achieved recognition accuracy rates of 91.59%/0.97%, 97.33%/0.70%, and 99.97%/0.03%. Our schemes are significantly better than existing channel solutions, with stable and robust recognition performance, and excellent generalization performance in DEAP-based recognition tasks. This provides a new reference for the development of wearable and portable EEG sensor devices and promotes the practical application of emotion recognition systems.