BERT Layer Weighting Comparision with Short Text Classification
- Resource Type
- Conference
- Authors
- Jayakody, JRKC; Vidanagama, VGTN; Perera, Indika; Herath, HMLK
- Source
- 2023 IEEE 17th International Conference on Industrial and Information Systems (ICIIS) Industrial and Information Systems (ICIIS), 2023 IEEE 17th International Conference on. :163-168 Aug, 2023
- Subject
- Bioengineering
Communication, Networking and Broadcast Technologies
Components, Circuits, Devices and Systems
Computing and Processing
Fields, Waves and Electromagnetics
Photonics and Electrooptics
Power, Energy and Industry Applications
Robotics and Control Systems
Signal Processing and Analysis
Support vector machines
Machine learning algorithms
Text categorization
Bidirectional control
Transformers
Encoding
Classification algorithms
Embeddings
BERT
short text
classification
- Language
BERT [Bidirectional Encoder Representations from Transformers] was the first type of neural model which was used hugely with several downstream tasks specially with the generated embedding representation for text classification. But with the existing literature different layer combination was not tested with concatenation as well as mean averaging techniques to represent the features of a text record for short text classification performances. Moreover, it was not properly evaluated with different machine learning algorithms with short text. Therefore, this research work focused on using seven short text type datasets with BERT embedding to identify the best embedding representation and the best machine learning algorithm. Based on conducted experiment, usage of all layers with concatenation techniques was identified as the best embedding representation and bagging as well as support vector machine was identified as the best machine learning algorithm for short text classification. Performance improvements with bagging was more than 6–8% over the other algorithm and concatenation of all layers gave around 5–6 % performance improvements over the other techniques.