Implementation of focused crawler by using machine learning approach

Abstract

The rapid growth of web generated on the internet by millions of users poses many challenges for general purpose search engines (for example, scaling).Typically a general purpose search engine consists of three main parts, Crawler, Indexer and Query processing system. The crawlers of a general purpose search engine crawl every page. So problems arise when we need to retrieve only corresponding portion of the web, especially for a topic or a group of topic. Such requirement can be fulfilled by a domain specific crawler or focused crawler. Focused crawler crawls only those pages that are interested by the system. A focused crawler traverses the web selecting out relevant pages to a predefined topic and neglecting those out of concern. The focused crawler determines which portion of the web is relevant and which is not. That can be done by several machine learning approach used in text categorization. This thesis proposes a focused crawler by using neural network. It can be used to build general purpose domain specific search engine.