Abstract:
Text classification becomes more and more challenging due to a scarcity of
standardized labeled data in the Myanmar NLP domain. The majority of the existing
Myanmar research has relied on models of deep learning that significantly focus on
context-independent word embeddings, such as Word2Vec, GloVe, and fastText, in which
each word has a fixed representation irrespective of its context. Meanwhile,
context-based pre-trained language models such as BERT and RoBERTa recently
revolutionized the state of natural language processing. In this paper, the experiments are
conducted to enhance the performance of classification in sentiment analysis by utilizing
the transfer learning ability of RoBERTa. Existing pretrained model based works only
utilize the last output layer of RoBERTa and ignore the semantic knowledge in the
intermediate layers. This research explores the potential of utilizing RoBERTa
intermediate layers to enhance the performance of fine-tuning of RoBERTa. To show the
generality, Myanmar pretrained RoBERTa model (MyanBERTa)[1] and multilingual
pretrained model (XLM-roBERTa)[3] are also compared. The effectiveness and
generality of intermediate layers were proved and discussed in the experimental result.
