Page 1 of 1

Spiders and Bots

PostPosted: January 24th, 2011, 11:09 pm
by SnowyCheetah
I've seen websites like Google browsing the forum with their accounts. They seem to be labeled "bot" sometimes, and "spider" other times. What's the difference between the two, if any?

Re: Spiders and Bots

PostPosted: January 25th, 2011, 6:09 am
by Moka
There isn't much of a difference. Spiders download webpages and bots peruse them and execute a predefined function such as index the page for use in a search engine.

Re: Spiders and Bots

PostPosted: January 25th, 2011, 6:12 am
by SnowyCheetah
What do spiders do with downloaded webpages? o.o

Re: Spiders and Bots

PostPosted: January 25th, 2011, 6:36 am
by Moka
It can vary...a good example is the web archive spider that downloads webpages to archive them: http://www.archive.org/web/web.php

Re: Spiders and Bots

PostPosted: January 25th, 2011, 7:51 am
by SnowyCheetah
Ok. And finally, crawlers?

Re: Spiders and Bots

PostPosted: January 25th, 2011, 9:43 am
by Moka
I think crawlers are just a name for bots and spiders that "crawl" the web by following links on the web page itself.

They are definitely loose terms, and some programmers define them differently than others. Their meaning is somewhat lost and people use them interchangeably. Personally, I call everything a bot. :]