Implementation of Web Crawler on Client Side

Abstract

Web Crawler forms the back-bone of applications that facilitate Web information retrieval. Generic crawlers and search engines are like public libraries: they try to cater to everyone, and do not specialize in specific areas. In this paper, we have presented the architecture and implementation details of our crawling system which can be deployed on the client machine to browse the web concurrently and autonomously. We have implemented the Crawler that can crawl and index pages from any Website. This demonstrates the working of a web crawler developed in java. Finally, Our Crawler will allow us to launch crawling process at maximum speeds without concern for the state of servers and usual network latency time.