Probing Statistical Representations for End-to-End ASR
- Resource Type
- Conference
- Authors
- Ollerenshaw, Anna; Jalal, Md Asif; Hain, Thomas
- Source
- 2023 31st European Signal Processing Conference (EUSIPCO) European Signal Processing Conference (EUSIPCO), 2023 31st. :401-405 Sep, 2023
- Subject
- Signal Processing and Analysis
Analytical models
Adaptation models
Europe
Signal processing
Transformers
Indexes
Task analysis
speech recognition
end-to-end
cross domain
transformer
analysis
language modelling
- Language
- ISSN
- 2076-1465
End-to-End automatic speech recognition (ASR) models aim to learn a generalised speech representation to perform recognition. In this domain there is little research to analyse internal representation dependencies and their relationship to modelling approaches. This paper investigates cross-domain language model dependencies within transformer architectures using SVCCA and uses these insights to identify critical parameters and improve recognition performance. It was found that specific neural representations within the transformer layers exhibit correlated behaviour which is related to recognition performance. Altogether, this work provides analysis of the modelling approaches affecting contextual dependencies and ASR performance, and can be used to create or adapt better performing End- to-End ASR models without the requirement for hyperparameter optimisation, and also for downstream tasks.