Multi-fidelity Matryoshka neural networks for constrained IoT devices
- Resource Type
- Authors
- Bart Dhoedt; Tim Verbelen; Elias De Coninck; Sam Leroux; Bert Vankeirsbilck; Steven Bohez; Pieter Simoens
- Source
- Ghent University Academic Bibliography
2016 International Joint Conference on Neural Networks
IJCNN
- Subject
- Sequence
Theoretical computer science
Technology and Engineering
Artificial neural network
Contextual image classification
Time delay neural network
Computer science
Distributed computing
media_common.quotation_subject
Fidelity
Block matrix
02 engineering and technology
010501 environmental sciences
01 natural sciences
020204 information systems
0202 electrical engineering, electronic engineering, information engineering
Memory footprint
IBCN
Nervous system network models
0105 earth and related environmental sciences
media_common
- Language
Using deep neural networks on resource constrained devices is a trending topic in neural network research. Various techniques for compressing neural networks have been proposed that allow evaluating a large neural network on a device with limited memory and processing power. These approaches usually generate a single compressed student network based on a larger teacher network. In some cases a more dynamic trade-off may be desired. In this paper we trained a sequence of increasingly large networks where each network is constrained to contain the unmodified features of all smaller networks. The weight matrix of the largest network has submatrices that correspond to the weight matrices of each of the smaller networks. This technique allows us to keep the parameters of several networks in memory while having the same memory footprint as the single largest network. A trade-off between accuracy and speed can be made at runtime. The proposed approach is validated on two image classification tasks running on a real-world Internet-of-Things (IoT) device.