A Study on Web Crawlers and Crawling Algorithms

Abstract

Making use of search engines is most popular Internet task apart from email. Currently, all major search engines employ web crawlers because effective web crawling is a key to the success of modern search engines. Web crawlers can give vast amounts of web information possible to explore the web entirely by humans. Therefore, crawling algorithms are crucial in selecting the pages that satisfy the users’ needs. Crawling cultural and/or linguistic specific resources from the borderless Web raises many challenging issues. This paper will review various web crawlers used for searching the web while also exploring the use of various algorithms to retrieve web pages.