Create a robots.txt file

Default -  All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank for none)
Specific Search Robots:	Google	Same as Default Allowed Refused   googlebot
	MSN Search	Same as Default Allowed Refused   msnbot
	Yahoo	Same as Default Allowed Refused   yahoo-slurp
	Ask/Teoma	Same as Default Allowed Refused   teoma
	Cuil	Same as Default Allowed Refused   twiceler
	GigaBlast	Same as Default Allowed Refused   gigabot
	Scrub The Web	Same as Default Allowed Refused   scrubby
	DMOZ Checker	Same as Default Allowed Refused   robozilla
	Nutch	Same as Default Allowed Refused   nutch
	Alexa/Wayback	Same as Default Allowed Refused   ia_archiver
	Baidu	Same as Default Allowed Refused   baiduspider
	Naver	Same as Default Allowed Refused   naverbot, yeti
Specific Special Bots:	Google Image	Same as Default Allowed Refused   googlebot-image
	Google Mobile	Same as Default Allowed Refused   googlebot-mobile
	Yahoo MM	Same as Default Allowed Refused   yahoo-mmcrawler
	MSN PicSearch	Same as Default Allowed Refused   psbot
	SingingFish	Same as Default Allowed Refused   asterias
	Yahoo Blogs	Same as Default Allowed Refused   yahoo-blogs/v3.9
Restricted Directories:	The path is relative to root and must contain a trailing "/"

Now, copy and paste this text into a blank text file
called "robots.txt" (don't forget the "s" on the end of
"robots") and put it in your root directory. Like all other
files on your server, make sure its permissions are set so that
visitors (such as search engines) can read it.