How to Control

Search Engine Robots

Wouldn't it be nice to be able to leave some code in your web site to tell

the search engine spider crawlers to make your site number one? Unfortunately a

robots.txt file or robots meta tag won't do that, but they can help the crawlers

to index your site better and block out the unwanted ones.

First a little definition explaining:

Search Engine Spiders or Crawlers - A web crawler (also

known as web spider) is a program which browses the World Wide Web in a

methodical, automated manner. Web crawlers are mainly used to create a copy of

all the visited pages for later processing by a search engine, that will index

the downloaded pages to provide fast searches.

A web crawler is one type of bot, or software agent. In general, it starts

with a list of URLs to visit. As it visits these URLs, it identifies all the

hyperlinks in the page and adds them to the list of URLs to visit, recursively

browsing the Web according to a set of policies.

Robots.txt - The robots exclusion standard or

robots.txt protocol is a convention to prevent well-behaved web spiders and

other web robots from accessing all or part of a website. The information

specifying the parts that should not be accessed is specified in a file called

robots.txt in the top-level directory of the website.

The robots.txt protocol is purely advisory, and relies on the cooperation of

the web robot, so that marking an area of your site out of bounds with

robots.txt does not guarantee privacy. Many web site administrators have been

caught out trying to use the robots file to make private parts of a website

invisible to the rest of the world. However the file is necessarily publicly

available and is easily checked by anyone with a web browser.

The robots.txt patterns are matched by simple substring comparisons, so care

should be taken to make sure that patterns matching directories have the final

'/' character appended: otherwise all files with names starting with that

Meta Tag - Meta tags are used to provide structured data about

data.

In the early 2000s, search engines veered away from reliance on Meta tags, as

many web sites used inappropriate keywords, or were keyword stuffing to obtain

any and all traffic possible.

Some search engines, however, still take Meta tags into some consideration

when delivering results. In recent years, search engines have become smarter,

penalizing websites that are cheating (by repeating the same keyword several

times to get a boost in the search ranking). Instead of going up rankings, these

websites will go down in rankings or, on some search engines, will be kicked off

of the search engine completely.

Index a site - The act of crawling your site and gathering

information.

How can the robots.txt file and meta tag help you?

In the robots.txt you can tell the harmful 'web crawlers' to leave your web

site alone, and give helpful hints to the ones you want to crawl your site.

Here is an example on how to disallow a web crawler to search your site:

# this identifies the wayback machine


User-agent: ia_archiver


Disallow: /

ia_archiver is the crawler name for the wayback machine that you may have

heard of, and the / after disallow tells ai_archiver not to index any of your

site. The # allows you to write comments to yourself so you

can keep track of what you typed.

Type the above three lines into notepad from your computer and save it to the

root directory of your web site as robots.txt. Web crawlers look for this

document first at a web site before doing anything else. This helps the

crawler to do its job, and helps the web site owner tell the spider what to do.

Say for instance you have some data that you don't want the crawlers to see.

(Like duplicate content for other browser referrer pages) You can deter

crawlers from indexing the 'duplicate' directory by typing this into your

robots.txt file.

Or if you would like to have the robots.txt file created for you, visit

www.rietta.com/robogen. To validate

your robots.txt file to make sure it works properly you can visit

www.searchengineworld.com/cgi-bin/robotcheck.cgi

User-agent: *


Disallow: /duplicate/

The * after user-agent says that this action applies to all crawlers and

/duplicate/ after disallow tells all crawlers to ignore this directory and not

search it. For each user-agent and disallow line there must be a blank

space between them in order for it to function correctly. So this is how

you would create the above two commands into a robots.txt file:

# this identifies the wayback machine


User-agent: ia_archiver


Disallow: /

User-agent: *


Disallow: /duplicate/

One thing to note that is very important: Anyone can access the

robots.txt file of a site. So if you have information that you don't want

anyone to see don't include it into the robots.txt file. If the directory

that you don't want anyone to see is not linked to from your web site the

crawlers won't index it anyway.

An alternative to blocking indexing of your site is to put a meta tag into

the page. It looks like this:

You put this into the tag of your web page. This line tells the

robot crawlers not to index (search) the page and not to follow any of the

hyperlinks on the page. So as an example

tells the robots crawlers to not index the page, but follow the hyperlinks on

this page.

Did you know that Google has its own tag?

It looks like this:

This tells the Google robot crawler not to index the page, not to follow any of

the links, and not to keep from storing cached versions of your web site.

You will want this done if you update the content on your site frequently.

This prevents the web user from seeing outdated content that isn't refreshed

because of storage in the cache.

You can use the tag to specifically talk to Google's robots to avoid

complications or if you are optimizing your site for Google's search engine.

This concludes this month's article.

Until the next article have a great day!

Copyright © Michael Rock


(You have permission to copy this article as long as it remains intact with the

author's byline)




Web development contractor (Web Design and Hosting)


Internet Presence


www.TheInternetPresence.com

The owner of this registered company

has over twenty years experience with DOS, windows business applications, numerous

programming languages, artistic development, and web design. Other areas of

interest include web marketing, web promoting, and business marketing and

development. After the persuasion of those praising his work, he decided to go

into business himself and highly suggests everyone else to do the same.




Internet Presence was founded in 2003

from a desire to become independent. Less than 1 year later Internet Presence

has had accounts in three different states ranging from a locally owned auto

collision repair shop to a glass packaging industry that sells its product

worldwide.