Text Compression and Prediction Using Language Model

Abstract

Language model in Natural Language Processing is one of the most important fields carried out in the world of Artificial Intelligence. The purpose of this paper is to express about stipulated abbreviation method and statistical language model in Natural Language Processing. The system solves the problem of improving the efficiency of natural language text input under degraded conditions by taking advantage of the informational redundancy in natural language. It takes advantage of the duality between prediction and compression. It allows the user to enter the English text and text in compressed form, using a simple stipulated abbreviation method. The system decodes the abbreviated text by using statistical language model. This paper is typically based on processing of sentences in English between the computer and the user. This is implemented by using C# language, Component One Software and Microsoft access database.