define Crawler

Definition:

Automated programs in search engines that gather web site listings by automatically crawling the web. A search engine’s crawler (also called a spider or robot) “reads” page text contents and web page coding, and also follows links to other hyperlinked pages on the web pages it crawls. A crawler makes copies of the web pages found and stores these in the search engine’s index, or database.