Abstract:
Making use of search engines is most popular
Internet task apart from email. Currently, all major
search engines employ web crawlers because effective
web crawling is a key to the success of modern search
engines. Web crawlers can give vast amounts of web
information possible to explore the web entirely by
humans. Therefore, crawling algorithms are crucial in
selecting the pages that satisfy the users’ needs.
Crawling cultural and/or linguistic specific resources
from the borderless Web raises many challenging
issues. This paper will review various web crawlers
used for searching the web while also exploring the use
of various algorithms to retrieve web pages.
