Nov
5
Written by:
Project Complete
11/5/2009 12:36 PM
Though we're almost a week past Halloween, it still seems appropriate to address web crawlers or web spiders. Like the insect of the same name, some are dangerous, but most are harmless. Sometimes you want them around, but most of the time we're glad we don't see them.
Simply put (which is how most of these blogs are), a web crawler is just a software program that crawls around the web methodically and automatically. Other terms for web crawlers are ants, automatic indexers, worms, web robots (or "bots") or web scutter. Whatever you call them, their function is to gather information. You'll be glad of them if you have a website that you want the search engines to find because search engines like Google, Yahoo or Ask, use them to index pages to provide fast searches. They are used for maintenance tasks such as checking links or validating HTML code. They "read" the visible text (they can't read graphics or even words on graphics), and keyword rich meta tags.
The web crawlers start with a list of URLs or web addresses (like www.projcomplete.com) to visit. Based on the information they find, the web crawlers include the website in the search engine's database, and is ranked. If a site is experiencing heavy traffic or is experiencing technical difficulties, the web crawler may be programmed to re-visit the site again. The bad spiders glean specific types of information from web pages, like email addresses, which are used by spammers. So I guess it isn't the spiders that are bad but the owners of the spiders.
Web crawlers are the driving force behind search engine optimization companies and why they can help companies get to the top of the natural or organic search results list. (One statistic I found said that 93% of Internet users never look beyond the first few results.) They find out the most popular keywords for your business, build those keywords into your company's website, put those words in all the right places, the right number of occurrences, etc. Then they find related companies who will link to your site. One way links to your site will create more traffic, causing the search engine company to rank your company's site higher than another company.
So don't worry about those creepy, crawly web crawlers. They're just doing their job, and will mean better results when you Google.