Month: October 2007

ia_archiver :: The Alexa ™ Search Engine Spider

The ia_archiver is the spider used by the Alexa Search Engine The Alexa crawler (robot) uses a web-wide crawl strategy. It starts with a list of known URLs from across the entire Internet. Using this list it then fetches all local links found as it goes. The ia_archiver spider will obey any robots exclusion protocol …


Search Engine Spider :: GoogleBot

1. Name of Spider: GoogleBot is the spider of the Google Search Engine. It comes in two forms: Deepbot and Freshbot. 2. What’s the difference between Deepbot and Freshbot? Freshbot can be though of as an indexing spider. It has a quick rummage of a site and then indexes pages on the site. Deepbot checks …