Learning Low-Rank Structured Sparsity in Recurrent Neural Networks
- Resource Type
- Conference
- Authors
- Wen, Weijing; Yang, Fan; Su, Yangfeng; Zhou, Dian; Zeng, Xuan
- Source
- 2020 IEEE International Symposium on Circuits and Systems (ISCAS) Circuits and Systems (ISCAS), 2020 IEEE International Symposium on. :1-4 Oct, 2020
- Subject
- Components, Circuits, Devices and Systems
Recurrent neural networks
Computer architecture
Sparse representation
Road transportation
Hardware
Logic gates
Recurrent Neural Network
Structured Sparsity
Low-rank Approximation
- Language
- ISSN
- 2158-1525
Acceleration and wide deployability in deeper recurrent neural network is hindered by high demand for computation and memory storage on devices with memory and latency constraints. In this work, we propose a novel regularization method to learn hardware-friendly sparse structures for deep recurrent neural networks. Considering the consistency of dimension in continuous time units in recurrent neural networks, low-rank structured sparse approximations of the weight matrices are learned through the regularization without dimension distortion. Our method is architecture agnostic and can learn compact models with higher degree of sparsity than the state-of-the-art structured sparsity learning method. The structured sparsity rather than random sparsity also facilitates the hardware implementation. Experiments on language modeling of Penn TreeBank dataset show that our approach can reduce the parameters of stacked recurrent neural network model by over 90% with less than 1% perplexity loss. It is also successfully evaluated on larger highway neural network model with word2vec dataset like enwik8 and text8 using only 20M weights.