LMCodec: A Low Bitrate Speech Codec with Causal Transformer Models
- Resource Type
- Conference
- Authors
- Jenrungrot, Teerapat; Chinen, Michael; Kleijn, W. Bastiaan; Skoglund, Jan; Borsos, Zalan; Zeghidour, Neil; Tagliasacchi, Marco
- Source
- ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2023 - 2023 IEEE International Conference on. :1-5 Jun, 2023
- Subject
- Bioengineering
Communication, Networking and Broadcast Technologies
Computing and Processing
Signal Processing and Analysis
Speech codecs
Convolutional codes
Uncertainty
Speech coding
Vector quantization
Bit rate
Predictive models
speech coding
Transformers
self-supervised learning
generative adversarial networks
- Language
- ISSN
- 2379-190X
We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test was conducted and shows that the quality is comparable to reference codecs at higher bitrates. Example audio is available at https://mjenrungrot.github.io/chrome-media-audio-papers/publications/lmcodec.