Computer programs, also called 'robots' or 'crawlers', that are used by search engines to catalogue the World Wide Web into huge databases. They obtain new pages, update known pages, and delete obsolete ones. Due to the size of the Internet and the time taken to 'sweep' it results are often out of date.
