Noise Block Cleaning and Main Content Block Extraction from Dynamic Web Page

Abstract

Web Information Extraction systems becomes more complex and time-consuming. Web page contains many informative blocks and noise blocks. Noise blocks are navigational elements, templates and advertisements that are not the main content blocks of the web page; it can be defined noisy blocks or boilerplate text. This boilerplate text typically is not related to the main content, may deteriorate search precision and thus needs to be detected properly. This paper proposes a Web Page cleaning and main content block extraction approach and purposes of improving the accuracy and efficiency of web content mining. The system uses structural features and the shallow text features as such as number of words, link density, and average word length can be used to classify the main content or boilerplate text from the web page. And then the system extracts main content block using three parameters such as Title keyword, Keyword Frequency based Block selection and position features. The relevant content blocks are identified as the high important level by similarity of block contents to other blocks. Experiments show that Web Page cleaning based on shallow features lead to more accurate and efficient classification results for boilerplate or other content than existing approaches.