A spider is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "crawler" or a "bot."
source for that excerpt:
http://whatis.techtarget.com/definition/spiderfurther reading at:
https://en.wikipedia.org/wiki/Web_crawler
