What is a Web Crawler? - Social Media GlossarySaturday, May 28, 2005
A web crawler is a tool used by a search engine to browse and index websites. They are also known as web bots and spiders.
Web crawlers are generally automated and comprised of complex algorithms that browse a site for new updates and content information.
Sites that are updated more frequently are typically crawled more quickly, which is why having a blog hosted on your website can be beneficial for SEO. The words in your blog posts count as keywords to your site and the more frequently you update, the more frequently you are indexed in Google with your new content.