Deep Dimension Reduction for Supervised Representation Learning
- Resource Type
- Periodical
- Authors
- Huang, J.; Jiao, Y.; Liao, X.; Liu, J.; Yu, Z.
- Source
- IEEE Transactions on Information Theory IEEE Trans. Inform. Theory Information Theory, IEEE Transactions on. 70(5):3583-3598 May, 2024
- Subject
- Communication, Networking and Broadcast Technologies
Signal Processing and Analysis
Dimensionality reduction
Representation learning
Estimation
Vectors
Linear programming
Data models
Covariance matrices
Conditional independence
distance covariance
f-divergence
nonparametric estimation
neural networks
- Language
- ISSN
- 0018-9448
1557-9654
The goal of supervised representation learning is to construct effective data representations for prediction. Among all the characteristics of an ideal nonparametric representation of high-dimensional complex data, sufficiency, low dimensionality and disentanglement are some of the most essential ones. We propose a deep dimension reduction approach to learning representations with these characteristics. The proposed approach is a nonparametric generalization of the sufficient dimension reduction method. We formulate the ideal representation learning task as that of finding a nonparametric representation that minimizes an objective function characterizing conditional independence and promoting disentanglement at the population level. We then estimate the target representation at the sample level nonparametrically using deep neural networks. We show that the estimated deep nonparametric representation is consistent in the sense that its excess risk converges to zero. Our extensive numerical experiments using simulated and real benchmark data demonstrate that the proposed methods have better performance than several existing dimension reduction methods and the standard deep learning models in the context of classification and regression.